clairemcginty opened a new issue, #16781: URL: https://github.com/apache/iceberg/issues/16781
I'm running Iceberg reads via [Apache Beam](https://github.com/apache/beam)'s iceberg [extension](https://github.com/apache/beam/tree/master/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg). Beam, and my project, are on Avro 1.11. When executing an Iceberg read, I hit the following runtime exception: ``` Exception in thread "sbt-bg-threads-1" java.lang.NoSuchMethodError: 'org.apache.avro.LogicalTypes$TimestampNanos org.apache.avro.LogicalTypes.timestampNanos()' at org.apache.iceberg.avro.TypeToSchema.<clinit>(TypeToSchema.java:50) at org.apache.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:64) at org.apache.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:59) at org.apache.iceberg.GenericManifestFile.<clinit>(GenericManifestFile.java:42) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) at org.apache.iceberg.common.DynConstructors$Ctor.newInstanceChecked(DynConstructors.java:51) at org.apache.iceberg.common.DynConstructors$Ctor.newInstance(DynConstructors.java:64) at org.apache.iceberg.avro.InternalReaders$PlannedStructLikeReader.reuseOrCreate(InternalReaders.java:67) at org.apache.iceberg.avro.InternalReaders$PlannedStructLikeReader.reuseOrCreate(InternalReaders.java:42) at org.apache.iceberg.avro.ValueReaders$PlannedStructReader.read(ValueReaders.java:1025) at org.apache.iceberg.avro.InternalReader.read(InternalReader.java:107) at org.apache.iceberg.avro.NameMappingDatumReader.read(NameMappingDatumReader.java:57) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:263) at org.apache.avro.file.DataFileStream.next(DataFileStream.java:248) at org.apache.iceberg.io.CloseableIterator$1.next(CloseableIterator.java:55) at org.apache.iceberg.relocated.com.google.common.collect.Iterators.addAll(Iterators.java:370) at org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:332) at org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:261) at org.apache.iceberg.ManifestLists.read(ManifestLists.java:42) at org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:185) at org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:211) at org.apache.iceberg.DataTableScan.doPlanFiles(DataTableScan.java:68) at org.apache.iceberg.SnapshotScan.planFiles(SnapshotScan.java:139) at org.apache.beam.sdk.io.iceberg.ScanSource.wholeTableReadTask(ScanSource.java:73) ... ``` This is happening because `timestamp-nanos` wasn't added to Avro until 1.12 (compare [1.11](https://github.com/apache/avro/blob/release-1.11.5/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L187-L197) types vs [1.12](https://github.com/apache/avro/blob/release-1.12.1/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L196-L208)). Upgrading Avro versions can be a nontrivial operation for various reasons - I was wondering if Iceberg would accept a change that gates the static initialization of `TIMESTAMP_NANO_SCHEMA` and `TIMESTAMPTZ_NANO_SCHEMA` behind an Avro version check. There's precedent for this in [parquet-java](https://github.com/apache/parquet-java/blob/apache-parquet-1.17.1/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L264-L266), however I'm not sure if Iceberg has made any commitments about Avro backwards-compatibility/if it's something that would be painful to maintain in the future. Let me know what you think; I have a proof of concept [here](https://github.com/apache/iceberg/compare/main...clairemcginty:iceberg:avro-1.11-schema-compat?expand=1) that works. thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
