zhztheplayer opened a new pull request, #11117: URL: https://github.com/apache/incubator-gluten/pull/11117
This is to fix result mismatch issues when Gluten is reading Parquet file written by old Spark versions (< 3.0.0) or with legacy datetime rebase mode (`spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY` / `spark.sql.parquet.int96RebaseModeInRead=LEGACY`). Test case `Column DEFAULT value support with Delta Lake, positive tests` which is being added in https://github.com/apache/incubator-gluten/pull/11107 will cover this fix. Before: ``` "000[2-12-30]" did not equal "000[1-01-01]" ScalaTestFailureLocation: org.apache.spark.sql.delta.DeltaColumnDefaultsInsertSuite at (DeltaInsertIntoTableSuite.scala:829) Expected :"000[1-01-01]" Actual :"000[2-12-30]" ``` After: ``` 15:22:23.853 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Scan parquet spark_catalog.default.t4[QueryId=43], due to: - Detected unsupported metadata in parquet files: Legacy timezone found. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
