kinolaev commented on code in PR #17196:
URL: https://github.com/apache/iceberg/pull/17196#discussion_r3613840467
##########
core/src/main/java/org/apache/iceberg/data/avro/DataReader.java:
##########
@@ -49,21 +49,37 @@ public static <D> DataReader<D> create(
public static <D> DataReader<D> create(
org.apache.iceberg.Schema expectedSchema, Schema readSchema,
Map<Integer, ?> idToConstant) {
- return new DataReader<>(expectedSchema, readSchema, idToConstant);
+ return create(expectedSchema, readSchema, idToConstant, true);
+ }
+
+ public static <D> DataReader<D> create(
Review Comment:
Partition schemas in manifests are affected by the new default. I have
pinned `legacyTimestampMapping=true` for manifest schemas that include
partitions in 5af496f1e5b9490ada1d8258940e7275f3df71a8.
For future safety, I also pinned `legacyTimestampMapping=true` for static
internal schemas that don't have any timestamps for now -
0c640180ff7bde7e2ea9d577ed9e372e96d3c379.
There are a few more places in the core library where AvroSchemaUtil is used:
- BuildAvroProjection
- GenericAvroReader
- IcebergEncoder/IcebergDecoder
As the next step I'll look at those. And then I'll try to add
`AVRO_TIMESTAMP_ENCODING_LEGACY_MODE` table property.
The final goal is:
- methods for working with external Avro-encoded data must follow the Avro
spec by default
- methods for working with Avro-encoded table data/delete files must follow
the Iceberg spec by default, with an option to switch to the Avro spec using
the table property
- methods for working with table manifests must follow the Iceberg spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
