[
https://issues.apache.org/jira/browse/FLINK-30548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
heyu dou updated FLINK-30548:
-----------------------------
Description:
When the avro schema changes, if the downstream uses the old schema to read the
data generated by the new schema.
The buffer pos of AvroDeserializationSchema.decoder will not be reset.
This will result in misaligned reads.
Because AvroDeserializationSchema.decoder is reuse and not reset buffer pos.
The next read should start from the latest pos.
[https://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java#L165]
This line should be changed to :
{code:java}
this.decoder = DecoderFactory.get().directBinaryDecoder(inputStream, null);
{code}
was:
When the avro schema changes, if the downstream uses the old schema to read the
data generated by the new schema.
The buffer pos of AvroDeserializationSchema.decoder will not be reset.
This will result in misaligned reads.
Because AvroDeserializationSchema.decoder is reuse and not reset buffer pos.
The next read should start from the latest pos.
[https://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java#L165]
This line should be changed to :
{code:java}
this.decoder = DecoderFactory.get().directBinaryDecoder(inputStream, null);
{code}
> In some cases AvroDeserializationSchema buffer is not reset
> -----------------------------------------------------------
>
> Key: FLINK-30548
> URL: https://issues.apache.org/jira/browse/FLINK-30548
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Reporter: heyu dou
> Priority: Major
>
> When the avro schema changes, if the downstream uses the old schema to read
> the data generated by the new schema.
> The buffer pos of AvroDeserializationSchema.decoder will not be reset.
> This will result in misaligned reads.
> Because AvroDeserializationSchema.decoder is reuse and not reset buffer pos.
> The next read should start from the latest pos.
> [https://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java#L165]
> This line should be changed to :
> {code:java}
> this.decoder = DecoderFactory.get().directBinaryDecoder(inputStream, null);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
