Ycallaer opened a new issue, #14811:
URL: https://github.com/apache/iceberg/issues/14811
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
Hi,
We are running Kafka Connect with Apache Iceberg 1.10.0 ( but this also
fails in 1.9.2) and after a while we see the connect instance throwing the
following error
`{"debug_level":"WARN","debug_timestamp":"2025-12-10
11:29:54,670","debug_thread":"task-thread-iceberg-sink-connector-epe-log-topics-v1-2","debug_file":"ErrorHandlers.java",
"debug_line":"222","debug_message":"Unable to parse error
response","debug_stacktrace":"java.io.UncheckedIOException:
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Invalid':
was expecting (JSON String, Number, Array, Object or token 'null', 'true' or
'false')
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION`
disabled); line: 1, column: 8]
at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:106)
at
org.apache.iceberg.rest.responses.ErrorResponseParser.fromJson(ErrorResponseParser.java:71)
at
org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.parseResponse(ErrorHandlers.java:220)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:210)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:336)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:297)
at org.apache.iceberg.rest.BaseHTTPClient.get(BaseHTTPClient.java:77)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:375)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:399)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:99)
at org.apache.iceberg.rest.RESTCatalog.loadTable(RESTCatalog.java:106)
at
org.apache.iceberg.connect.data.IcebergWriterFactory.createWriter(IcebergWriterFactory.java:59)
at
org.apache.iceberg.connect.data.SinkWriter.lambda$writerForTable$3(SinkWriter.java:139)
at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
at
org.apache.iceberg.connect.data.SinkWriter.writerForTable(SinkWriter.java:138)
at
org.apache.iceberg.connect.data.SinkWriter.routeRecordDynamically(SinkWriter.java:124)
at org.apache.iceberg.connect.data.SinkWriter.save(SinkWriter.java:83)
at java.base/java.util.ArrayList.forEach(Unknown Source)
at org.apache.iceberg.connect.data.SinkWriter.save(SinkWriter.java:68)
at org.apache.iceberg.connect.channel.Worker.save(Worker.java:124)
at
org.apache.iceberg.connect.channel.CommitterImpl.save(CommitterImpl.java:163)
at
org.apache.iceberg.connect.IcebergSinkTask.put(IcebergSinkTask.java:83)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:624)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:349)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:248)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:217)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:249)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:304)
at
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$7(Plugins.java:341)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token
'Invalid': was expecting (JSON String, Number, Array, Object or token 'null',
'true' or 'false')
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION`
disabled); line: 1, column: 8]
at
com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2596)
at
com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2622)
at
com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2630)
at
com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:830)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:3017)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2995)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser._matchToken(ReaderBasedJsonParser.java:2760)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2035)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:780)
at
com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:5058)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4961)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3887)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3855)
at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:104)
... 33 more
"}
{"debug_level":"ERROR","debug_timestamp":"2025-12-10
11:29:54,670","debug_thread":"task-thread-iceberg-sink-connector-epe-log-topics-v1-2","debug_file":"WorkerSinkTask.java",
"debug_line":"656","debug_message":"WorkerSinkTask{id=iceberg-sink-connector-epe-log-topics-v1-2}
Task threw an uncaught and unrecoverable exception. Task is being killed and
will not recover until manually restarted. Error: Forbidden: Invalid
Token","debug_stacktrace":"org.apache.iceberg.exceptions.ForbiddenException:
Forbidden: Invalid Token
at
org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:238)
at
org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:124)
at
org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:108)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:240)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:336)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:297)
at org.apache.iceberg.rest.BaseHTTPClient.get(BaseHTTPClient.java:77)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:375)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:399)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:99)
at org.apache.iceberg.rest.RESTCatalog.loadTable(RESTCatalog.java:106)
at
org.apache.iceberg.connect.data.IcebergWriterFactory.createWriter(IcebergWriterFactory.java:59)
at
org.apache.iceberg.connect.data.SinkWriter.lambda$writerForTable$3(SinkWriter.java:139)
at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
at
org.apache.iceberg.connect.data.SinkWriter.writerForTable(SinkWriter.java:138)
at
org.apache.iceberg.connect.data.SinkWriter.routeRecordDynamically(SinkWriter.java:124)
at org.apache.iceberg.connect.data.SinkWriter.save(SinkWriter.java:83)
at java.base/java.util.ArrayList.forEach(Unknown Source)
at org.apache.iceberg.connect.data.SinkWriter.save(SinkWriter.java:68)
at org.apache.iceberg.connect.channel.Worker.save(Worker.java:124)
at
org.apache.iceberg.connect.channel.CommitterImpl.save(CommitterImpl.java:163)
at
org.apache.iceberg.connect.IcebergSinkTask.put(IcebergSinkTask.java:83)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:624)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:349)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:248)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:217)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:249)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:304)
at
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$7(Plugins.java:341)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
"}`
This is a partial snippet of our Connect configuration
`{
"connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
"iceberg.catalog.scope": "all-apis",
"errors.log.include.messages": "true",
"iceberg.tables.evolve-schema-enabled": "true",
"tasks.max": "6",
"iceberg.catalog.security": "OAUTH2",
"iceberg.tables.dynamic-enabled": "true",
"iceberg.tables.auto-create-enabled": "true",
"confluent.topic.replication.factor": "1",
"iceberg.catalog.oauth2-server-uri":
"https://***.***.azuredatabricks.net/oidc/v1/token";,
"errors.log.enable": "true",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"iceberg.kafka.ssl.endpoint.identification.algorithm": "",
"iceberg.catalog.credential":
"${file:/tmp/iceberg-connect-catalog-credential/credential:iceberg_secret}",
"value.converter.enhanced.protobuf.schema.support": "true",
"iceberg.catalog.uri":
"https://***.***.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest";,
"iceberg.catalog.vended-credentials-enabled": "true",
"value.converter.schemas.enable": "false",
"name": "iceberg-sink-connector",
"errors.tolerance": "all",
"iceberg.catalog.type": "rest"
}`
For now our workaround is to periodically restart the connector, which then
retrieves a valid OAUTH token, which let's the connector continue with
processing.
Is there a way that the proper authentication flow could be handled in the
backend.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
