RyanSkraba commented on PR #18635: URL: https://github.com/apache/flink/pull/18635#issuecomment-1197281831
Hello! These are both good notes. I have some opinions, but I'd appreciate any ideas or feedback! There _might_ be compatibility concerns, although this is mitigated by two points: 1. The changed schema is a drop-in replacement: bytes serialised by the old schema can be deserialized transparently by the new schema and vice versa. 2. If the old schema was persisted somewhere (file or registry), and the user is using Avro schema resolution while reading, adding an explicit namespace won't cause any problems. In my experience, there will probably be at least one user doing some custom processing or storage of generated schemas, and we can't prove they won't be surprised by the change. That being said, we are definitely encountering users today that want to use the generated schema with a Avro Python SDK. > Should we add a test that verifies the generated schemas are compatible with python? I don't want to point fingers, but I think that this Java/Python interoperability failure is an Avro issue, and should be addressed there. It would be unreasonable (for example) for Flink to verify that it's not accidentally creating schemas that can't be read by the Rust SDK... I created AVRO-3591 for the next major release of Avro (and I am very motivated to keep the confidence in Avro's interoperability high). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
