kevinjqliu commented on issue #2663: URL: https://github.com/apache/iceberg-python/issues/2663#issuecomment-3481779377
> Basically Dremio only support timestamp types. However, they mark the actual parquet files with a utc-adjustment flag, which in turn pyiceberg decide to interpurt as a TimestampTz, even though the concrete schema type is Timestamp. This is a pyiceberg bug, at least that's the way I see it. I believe you're referring to this logic. https://github.com/apache/iceberg-python/blob/8878b2c260bf62ad16ef6f8e8d3a6ae93103eff7/pyiceberg/io/pyarrow.py#L1387-L1390 I think you're right, we should use the `adjustToUtc` field to determine timestamp vs timestamptz as specified in the spec https://iceberg.apache.org/spec/#parquet and by the java implementation https://github.com/apache/iceberg/blob/94d9287b7a3b7474aa0776c315ec000fc97de329/parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java#L225-L230 > Thank you for bringing this up and for sharing the details, [@sheinbergon](https://community.dremio.com/u/sheinbergon). You’re right - in Dremio OSS 26.0.0, Iceberg tables created with timestamp columns may be recorded in the metadata as timestamptz, even though Dremio only supports timestamp. As you noted, this can cause compatibility issues with external readers. This might also be a Dremio issue too https://community.dremio.com/t/incorrect-timestamp-field-type-in-for-created-iceberg-table/13095/5 which was mentioned here as fixed https://community.dremio.com/t/incorrect-timestamp-field-type-in-for-created-iceberg-table/13095/7 @chrisqiqiu could you give it a try with the new Dremio release? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
