djouallah opened a new issue, #13429: URL: https://github.com/apache/gluten/issues/13429
### Description Gluten's native Iceberg write stores a Spark `TIMESTAMP` column (Iceberg `timestamptz`) as Parquet `Timestamp(isAdjustedToUTC=false, MICROS)`. The Iceberg spec requires `isAdjustedToUTC=true` for `timestamptz` (`timestamp` without zone is the `false` case). Footer of a data file written by Gluten (`created_by: parquet-cpp-velox version 0.0.0`), table column `129: SETTLEMENTDATE: optional timestamptz`: ``` optional int64 field_id=129 SETTLEMENTDATE (Timestamp(isAdjustedToUTC=false, timeUnit=microseconds, ...)); ``` Spark and Java Iceberg readers ignore the flag, so it goes unnoticed there. Stricter readers refuse the files: apache/iceberg-go fails compaction with `cannot promote timestamp to timestamptz`. Expected: `isAdjustedToUTC=true` for `timestamptz` columns, as vanilla Spark writes them. Setup: gluten-velox-bundle-spark4.1_2.13-linux_amd64-1.8.0-SNAPSHOT (nightly), Spark 4.1.1, Iceberg REST catalog, `INSERT`/`MERGE` from dbt-spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
