sheinbergon commented on issue #2663: URL: https://github.com/apache/iceberg-python/issues/2663#issuecomment-3487995736
@kevinjqliu if you notie who opened this thread in the dremio community forum, you see he and I share the same avatar 😉 There was a bug in dremio where they wrote an incorrect type to the schema, they fixed it in 26.0.5, but nonetheless, the bug in the `schema_conversion.py` module of PyIceberg remains, as it causes the library to incorrectly interpert the schema and then fail on type promotion later down the line. If the column is a timestamp column (which is what Dremio supportts ATM) why should `adjust-to-utc` flag on a parquet file column cause it to be read differently? Dremio may or may not set this flag but it does mean to store a TZ-less timestamp. If the schema defines a TimestampType column, and data is a TimestampType column, I feel the library should not back itself corners such as these, especially if the gain is adding a utc indication for timestamps, as the user did not intended to include TZ awareness in the schema. That's why I preceive this issue as a Pyiceberg, rather than a Dremio one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
