clairemcginty commented on PR #16824: URL: https://github.com/apache/iceberg/pull/16824#issuecomment-4972584707
> @clairemcginty it's not entirely clear to me why we would want to make Iceberg 1.11.0+ work with Avro 1.11 given that Iceberg 1.11.0 already uses Avro 1.12.1. In fact, Iceberg has been on Avro 1.12.0 since [Iceberg 1.7.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.7.0/gradle/libs.versions.toml#L29). This sets a precedent for the project that such workarounds are acceptable to make things work with Avro 1.11 but what if other pieces of Avro 1.12.x don't work with Avro 1.11? Are we going to fix those places as well? Ultimately the root cause is that there's a dependency mismatch between Avro versions in the consuming project and I don't think Iceberg should be accommodating this by having a workaround in the code. hey @nastra, porting some of the discussion we were having on Slack to this PR: I understand the concern of not wanting to make a commitment to supporting an older version of Avro for an indefinite amount of time - however, Avro 1.11 is still quite widely used (for example, [Flink](https://github.com/apache/flink/blob/release-2.3.0/pom.xml#L151), [parquet-java](https://github.com/apache/parquet-java/blob/apache-parquet-1.17.1/pom.xml#L99), the [latest Spark 3.x release](https://github.com/apache/spark/blob/branch-3.5/pom.xml#L157), and [Apache Beam](https://github.com/apache/beam/blob/v2.75.0/sdks/java/extensions/avro/build.gradle#L39-L43) are all still on 1.11... although the Hadoop trunk branch is now on 1.12 its [latest release](https://github.com/apache/hadoop/blob/rel/release-3.5.0/hadoop-project/pom.xml#L65) uses 1.11, too). Because of the new logical types added between in 1.12, upgrading from 1.11 as an Avro data producer is a nontrivial task if your Avro-producing and Avro-consuming applications are at all decoupled from each other. On the other hand, as a library owner, outside of the new logical types, the core Avro library APIs have remained relatively stable between 1.11/1.12. There are some changes to the schema parser, and a few new constructors added to Schema, but backwards compatibility has mostly been maintained. Overall I do think that supporting existing Avro 1.11 would help increase iceberg adoption. I can try to ask a few more committers for their thoughts if that's what you recommend! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
