DanielLeens commented on issue #10405: URL: https://github.com/apache/seatunnel/issues/10405#issuecomment-4880124759
Thanks for adding the full stack trace. After rechecking the current Avro path, this now looks more like a genuine compatibility bug than a simple config mistake. The key detail is that SeaTunnel builds its own Avro schema from the SeaTunnel table schema and then uses that generated schema as both the writer and reader schema during deserialization. The generated nullable unions are emitted as `[null, <type>]`, while your producer schema uses `[<type>, null]`. For Avro binary data, union branch order is part of the encoding, so this mismatch can explain the `ArrayIndexOutOfBoundsException` you are seeing even after marking the fields nullable. So the current Kafka + Avro path is too strict for externally produced Avro payloads whose writer schema does not exactly match SeaTunnel's generated one. Flink succeeds here because it is deserializing against the producer-side schema, rather than rebuilding the schema in a different shape from the table definition alone. This issue is worth keeping open as a real bug in the current compatibility path. A useful fix direction would be one of these: 1. allow the Kafka Avro path to use the actual writer schema / schema-registry schema when available 2. or add an explicit external Avro schema option instead of relying only on the generated SeaTunnel schema 3. keep the generated-schema path only for cases where producer and consumer schema shape is known to match exactly We also labeled this as `help wanted`, since the reporter did not opt into a PR and this looks like a good contribution target. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
