johntomcat7408-cmyk opened a new pull request, #28746: URL: https://github.com/apache/flink/pull/28746
What is the purpose of the change The avro format returns a ProjectableDecodingFormat, but projection pushdown only works when the projected fields start at index 0; a projection such as [[1], [2]] fails with ArrayIndexOutOfBoundsException. The deserializer used the projected schema as both the writer and reader schema, so the positional Avro decoding reads the wrong fields. This passes the full physical schema as the writer schema and the projected schema as the reader schema, so Avro schema resolution reads only the projected fields. Brief change log AvroDeserializationSchema: support an optional writer schema and resolve the input with setSchema(writer) + setExpected(reader); add a forGeneric(Schema reader, Schema writer) overload. AvroFormatFactory#createRuntimeDecoder: use the projected schema as the reader schema and the full physical schema as the writer schema. Verifying this change AvroRowDataDeSerializationSchemaTest Does this pull request potentially affect one of the following parts: Dependencies (does it add or upgrade a dependency): no The public API, i.e., is any changed class annotated with @public(Evolving): no The serializers: no The runtime per-record code paths (performance sensitive): yes Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no The S3 file system connector: no Documentation Does this pull request introduce a new feature? no If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
