kinolaev commented on PR #17196: URL: https://github.com/apache/iceberg/pull/17196#issuecomment-6640967077
Hi @RussellSpitzer, with recent developments in v4 in mind, I've decided to cut this PR down to the minimal change we previously agreed on. Please give it a second chance. It's small now and shouldn't take long to review. The PR only extends Avro-to-Iceberg schema conversion and the Avro readers for Avro-native timestamps: - adds one public `AvroSchemaUtil.convert` overload with an `adjustToUtcDefault` parameter, which decides how to interpret `timestamp-*` without `adjust-to-utc`, - maps `local-timestamp-*` to `timestamp` and `timestamp_ns`, - lets the readers read `local-timestamp-*`. The PR doesn't touch the Iceberg-to-Avro conversion or the Avro writers, and it no longer adds table properties. It addresses just one specific need: it makes it possible to use `AvroSchemaUtil` when reading external Avro-encoded messages, for example, from Kafka topics. Reader support is added for consistency, so that `local-timestamp-*` columns can be read as the timestamp types they now convert to. It's in a separate commit, so we can drop it from this PR if you prefer. I decided to cut the PR down when I realized that the merged AMT proposal #16025 adds a `format_version` field only to manifest entries, so it can't be used at the data file level. Then I read two recent discussions about making Parquet the only supported format for [manifests](https://lists.apache.org/thread/l703q6mcyzbgkfczs91mj5bffk1k5pj4) and even for [data files](https://lists.apache.org/thread/5jfzckl0cc9z683kz9vfv41pmhrm1tdb). So, as you said before, the question of Avro encoding in v4 has become largely moot. Anyway, if Avro stays in the spec and anyone is interested in supporting Avro-native timestamps at the data or manifest level, this PR will save them a step. The previous version of the PR is saved in [a separate branch](https://github.com/kinolaev/iceberg/tree/feat-add-avro-logical-types-full), just in case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
