[
https://issues.apache.org/jira/browse/FLINK-24544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431204#comment-17431204
]
Peter Schrott edited comment on FLINK-24544 at 10/20/21, 12:59 PM:
-------------------------------------------------------------------
The underlying problem with deserialization of records with enums form Kafka &
schema registry lies in the initialization of {{GenericDatumReader}}:
Case Kafka & SR:
In {{AvroDeserializationSchema.java}} the {{GenericDatumReader}} is initialized
with {{writerSchema = null}} and {{readerSchema = the schma gained from table
ddl}}
-> When calling {{RegistryAvroDeserializationSchema.deserialize(.)}}
{{datumReader.setSchema()}} sets the attribute {{actual}} is set to the actual
avro schema, whereas {{expected}} is already set to {{readerSchema}}
-> The inequality of {{actual}} and {{expected}} causes the exception on
serializing as type of {{actual}} and {{expected}} do not match
--> Root of this is: the initialization of {{DeserializationSchema}} in
{{RegistryAvroFormatFactory.java}} uses the {{rowType}} && {{ }} when creating
the {{ConfluentRegistryAvroDeserializationSchema}}
Case FS:
In {{AvroInputFormat.java}} the {{GenericDatumReader}} is initialized with
{{writerSchema = null}} and {{readerSchema = null}}
-> This leads in initialization of {{DataFileStream}}, where
{{reader.getSchema(.)}} is called with the actual avro, to the fact that in the
{{GenericDatumReader}} attribute {{expected}} and {{actual}} is set to the
passed value
-> The avro schema is taken from file
-> The equality of {{actual}} and {{expected}} leads to the fact that
serialized data can be read from file
was (Author: peter.schrott):
The underlying problem with deserialization of records with enums form Kafka &
schema registry lies in the initialization of `GenericDatumReader`:
Case Kafka & SR:
In `AvroDeserializationSchema.java` the `GenericDatumReader` is initialized
with `writerSchema = null` and `readerSchema = the schma gained from table ddl`
-> When calling `RegistryAvroDeserializationSchema.deserialize(.)`
`datumReader.setSchema()` sets the attribute `actual` is set to the actual avro
schema, whereas `expected` is already set to `readerSchema`
-> The inequality of actual` and `expected` causes the exception on serializing
as type of `actual` and `expected` do not match
--> Root of this is: the initialization of `DeserializationSchema` in
`RegistryAvroFormatFactory.java` uses the `rowType` &
`ConfluentRegistryAvroDeserializationSchema.forGeneric` when creating the
`ConfluentRegistryAvroDeserializationSchema`
Case FS:
In `AvroInputFormat.java` the `GenericDatumReader` is initialized with
`writerSchema = null` and `readerSchema = null`
-> This leads in initialization of `DataFileStream`, where
`reader.getSchema(.)` is called with the actual avro, to the fact that in the
`GenericDatumReader` attribute `expected` and `actual` is set to the passed
value
-> The avro schema is taken from file
-> The equality of `actual` and `expected` leads to the fact that serialized
data can be read from file
> Failure when using Kafka connector in Table API with Avro and Confluent
> schema registry
> ----------------------------------------------------------------------------------------
>
> Key: FLINK-24544
> URL: https://issues.apache.org/jira/browse/FLINK-24544
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Formats (JSON, Avro, Parquet, ORC,
> SequenceFile), Table SQL / Ecosystem
> Affects Versions: 1.13.1
> Reporter: Francesco Guardiani
> Priority: Major
> Attachments: flink-deser-avro-enum.zip
>
>
> A user reported in the [mailing
> list|https://lists.apache.org/thread.html/re38a07f6121cc580737a20c11574719cfe554e58d99817f79db9bb4a%40%3Cuser.flink.apache.org%3E]
> that Avro deserialization fails when using Kafka, Avro and Confluent Schema
> Registry:
> {code:java}
> Caused by: java.io.IOException: Failed to deserialize Avro record.
> at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
> at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
>
> at
> org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
> at
> org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:113)
> at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
>
> at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
> Caused by: org.apache.avro.AvroTypeException: Found my.type.avro.MyEnumType,
> expecting union
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:308)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:86)
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:275)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:298)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.deserialize(RegistryAvroDeserializationSchema.java:81)
> at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:103)
> ... 9 more
> {code}
> Look in the attachments for a reproducer.
> Same data serialized to a file works fine (look the filesystem example in the
> reproducer)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
