kevinjqliu commented on issue #2663: URL: https://github.com/apache/iceberg-python/issues/2663#issuecomment-3488523385
ah i did not make that connection 😄 Ok i think i see whats going on here. From the [Dremio thread](https://community.dremio.com/t/incorrect-timestamp-field-type-in-for-created-iceberg-table/13095/5) > Thank you for bringing this up and for sharing the details, [@sheinbergon](https://community.dremio.com/u/sheinbergon). You’re right - in Dremio OSS 26.0.0, Iceberg tables created with timestamp columns may be recorded in the metadata as timestamptz, even though Dremio only supports timestamp. As you noted, this can cause compatibility issues with external readers. The issue is that Dremio incorrectly produced the metadata (in avro) which specified certain columns as timestamptz while the data files (in parquet) specified those columns as timestamp (without tz). If that is the case, I believe pyiceberg is doing the right thing by failing on "type promotion". Its using the metadata as the source of truth and expects the data files to have the same type, in this case timestamptz. We cannot process timestamptz as timestamp or vice versa since they are considered to be different data types. > If the column is a timestamp column (which is what Dremio supportts ATM) why should adjust-to-utc flag on a parquet file column cause it to be read differently? The spec uses `adjust-to-utc` as a signal for timestamp vs timestamptz. And consider these to be 2 different data types. See https://iceberg.apache.org/spec/#avro Furthermore, "Avro type annotation adjust-to-utc is an Iceberg convention; default value is false if not present." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
