kinolaev commented on code in PR #17196:
URL: https://github.com/apache/iceberg/pull/17196#discussion_r3622524521
##########
core/src/main/java/org/apache/iceberg/data/avro/DataReader.java:
##########
@@ -49,21 +49,37 @@ public static <D> DataReader<D> create(
public static <D> DataReader<D> create(
org.apache.iceberg.Schema expectedSchema, Schema readSchema,
Map<Integer, ?> idToConstant) {
- return new DataReader<>(expectedSchema, readSchema, idToConstant);
+ return create(expectedSchema, readSchema, idToConstant, true);
+ }
+
+ public static <D> DataReader<D> create(
Review Comment:
I added the `write.avro.local-timestamp.enabled` table property, and renamed
`legacyTimestampMapping` to `localTimestampEnabled` in Iceberg->Avro conversion
methods, and to `adjustToUtcDefault` in Avro->Iceberg conversion methods.
In `AvroSchemaUtil`, `IcebergEncoder` and `IcebergEncoder` both parameters
default to `true`, since these classes can be used with external Avro-encoded
data.
The `write.avro.local-timestamp.enabled` table property defaults to `false`.
When it is `true`, the produced Avro schema gets a new property
`adjust-to-utc-default=true` that is used by `DataWriter` and then by
`(Planned)DataReader`.
Where `AvroSchemaUtil` works with manifests, both parameters are always
pinned to `false`.
~~I'm not sure about the default value in the
`org.apache.iceberg.parquet.Parquet` reader, or if it should be configurable at
all. So, I added `avroLocalTimestampEnabled=false` to preserve the current
behavior.~~ I dropped the changes in the parquet module. It seems like
`ParquetReadSupport` isn't used in the Iceberg modules - `ReadBuilder` always
gets a `batchedReaderFunc`/`readerFunction`, so the `build` method terminates
early. The change would only break external code if someone uses
`Parquet.read().build()` without calling `createReaderFunc()` and expects
`Instant` instead of `LocalDateTime` for a field of type
`timestamp`/`timstamp_ns`, which I consider a bug.
@AnatolyPopov and @RussellSpitzer, I'm ready for and would appreciate your
next review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
