radai-rosenblatt commented on pull request #1333: URL: https://github.com/apache/avro/pull/1333#issuecomment-1011640589
Hey folks. I'd like to elaborate a little here about why the avro-util project exists. we can then see if involving upstream avro makes sense in anything. TL;DR - when using avro in a large, diverse ecosystem, supporting multiple versions of avro simultaneously across the ecosystem is required. there a few things that would make this work easier listed near the bottom. the main reasons are code/schema reuse and portable libraries. Linkein uses avro for encoding across a lot of its "data plane" (kafka being a good example). a lot of these payloads are "shared models" - data produced by service X that is consumed by services Y and Z over a kafka topic, or any other storage medium or rpc. as such all parties involved must have "the same schema". this is especially true for avro - which requires the exact writer schema on-hand for decoding - vs other serialization formats. the best way to guarantee all parties have the same schema is to have the schema(s) in question in their own library, imported by all parties. schemas are not the only thing shared here, code is as well: its much more convenient for developers to operate on generated POJO classes than it is to operate on generic records, and there are libraries who's top level APIs accept/return IndexedRecords, meaning they expose avro. given the sheer number of codebases involved, and that some constraints on avro are beyond the ability of an organization to control (dictated by requirements of 3rd party external libraries), its not feasible to align on a single version of avro across the organization, and individual codebases change their version of avro on their own schedules. and so, even maintaining a library of generated record classes, not to mention writing a "portable" library that accepts/returns IndexedRecords requires navigating a very wide range of runtime avro versions. we do so by combination of adapters (different implementations of "the same thing" for different avro versions) and tweaking the generated specific classes. its obviously unreasonable to expect the avro API to never change. however, there have been previous breaking changes that could have been done "nicer". some examples are below if youre curious. things the avro project could change to make maintaining avro-util easier: - document breaking changes to APIs and/or behaviors. currently we try "guessing" those by looking at commit logs but more often than not our users find them 1st - consider adding options to better support cross-version compatibility. this is especially painful in generated code where something as simple as not adding an @Overrides annotation to new methods immediately saves a surprising amount of compilation issues. having default implementations for new methods in the parent abstract classes would also help a great deal. - "phase in" breaking changes with a period of error logs? examples of past breaking changes that could have been less breaking: - the json format for union encoding was changed between 1.4 and 1.5 to use a fullname as discriminator instead of simple name. the parser could have been made to accept both forms for cases where the simple names are distinct to make this a less breaking change (we eventually wrote our own version of JsonDecoder that does exactly that) - in more recent memory (https://issues.apache.org/jira/browse/AVRO-2035) validation for default values was tightened - which is great. but some things like a default value of "true" (a json string) for a binary property could have been kept around initially (with a very nagging error printed at parse time) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
