kinolaev commented on PR #17196: URL: https://github.com/apache/iceberg/pull/17196#issuecomment-4975172209
> Could you clarify which related PRs are considered breaking here? All three PRs unconditionally add support for `local-timestamp-*` types to `SchemaToType`. This means the output of methods like `AvroSchemaUtil.toIcebergSchema` will change after an upgrade (long columns will become timestamps), which could break downstream pipelines. > Avro to Iceberg conversion should continue honoring adjust-to-utc To avoid this breaking change, I actually prefer having two separate modes: 1. Iceberg-specific: Ignores `local-timestamp-*` and honors the `adjust-to-utc` property. 2. Avro-native: Honors logical types and ignores the `adjust-to-utc` property. By the way, `local-timestamp-*` types are already supported in Flink `AvroSchemaConverter` and `AvroToRowDataConverter` when `legacyTimestampMapping` is set to `false`. I decided to adapt the same approach for `AvroSchemaUtil`. > Is this PR intentionally limited to an external schema-conversion API rather than end-to-end read/write support? Yes, it is intentionally limited. I recently started [a project to ingest data from Kafka using the Flink Dynamic Iceberg Sink](https://github.com/kinolaev/iceberg-flink-kafka). Kafka Connect serializes the data into Avro format, and Iceberg Flink converts the Avro `GenericRecord` to an Iceberg `Schema` and Flink `RowData`. While this seemed straightforward at first, I ran into an issue with timestamp handling. Since Flink already has an Avro-native mode, I'm only looking to add support for Avro-native mode in `AvroSchemaUtil` for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
