sheinbergon commented on issue #2663: URL: https://github.com/apache/iceberg-python/issues/2663#issuecomment-3453657468
This is related to how Dremio creates the parquet files, and how pyiceberg interperts them. See [this](https://community.dremio.com/t/incorrect-timestamp-field-type-in-for-created-iceberg-table/13095/2) communit thread in Dremio's community forum. Basically Dremio only support timestamp types. However, they mark the actual parquet files with a utc-adjustment flag, which in turn pyiceberg decide to interpurt as a TimestampTz, even though the concrete schema type is Timestamp. This is a pyiceberg bug, at least that's the way I see it. I think you should be able to work around it if you create a complementry table using pyiceberg with timestamptz columns, and copy data to it. I think I found the actual location in the pyiceberg source code where this manipulation happens, but have yet to open an issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
