stuartweir commented on issue #10217:
URL: https://github.com/apache/hudi/issues/10217#issuecomment-3716013650
Hey Hudi team, I am encountering an issue with deserializing Apache BEAM
Confluent topic (Avro deserializer and schema registry) messages with this
error:
```
Caused by: org.apache.kafka.common.errors.SerializationException: Error
deserializing key/value for partition TOPICNAME-0 at offset 110. If needed,
please seek past the record to continue consumption.
Caused by: org.apache.kafka.common.errors.SerializationException: Error
retrieving Avro unknown schema for id 147190
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
Unrecognized field "metadata" (class
io.confluent.kafka.schemaregistry.client.rest.entities.SchemaString), not
marked as ignorable (4 known properties: "references", "schema", "schemaType",
"maxId"])
at [Source: UNKNOWN; line: 1, column: 714] (through reference chain:
io.confluent.kafka.schemaregistry.client.rest.entities.SchemaString["metadata"])
```
The Confluent topic Schema:
```json
{
"doc": "TOPICNAME schema",
"fields": [
{
"default": "unknown",
"name": "uuid",
"type": [
"string",
"null"
]
},
{
"name": "foo_id",
"type": "string"
},
{
"name": "alert",
"type": "boolean"
},
{
"default": "unknown",
"name": "bazz",
"type": [
"string",
"null"
]
},
{
"default": "unknown",
"name": "fizz",
"type": [
"string",
"null"
]
},
{
"name": "inserted_at",
"type": "double"
},
{
"name": "buzz",
"type": "string"
},
{
"name": "bizz",
"type": "string"
},
{
"default": "unknown",
"name": "metadata",
"type": [
"string",
"null"
]
}
],
"name": "TOPICNAME",
"namespace": "com.de_beam",
"type": "record"
}
```
Hudi version being used is 0.15. However I noticed that Hudi 1.1.0 has not
updated kafka-schema-registry-client.
Adding this as per @xushiyan's request in the Hudi slack here:
https://apache-hudi.slack.com/archives/C4D716NPQ/p1765556228417399?thread_ts=1765388649.036399&cid=C4D716NPQ
This seems to be a critical issue at this point, at least for my team. If
Confluent's Schema Registry updates to a newer version for Debezium, this would
greatly affect our ability to continue to use Hudi long term without a lot of
effort to add custom Schema Converters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
