peterxcli opened a new pull request, #5048: URL: https://github.com/apache/datafusion-comet/pull/5048
## Which issue does this PR close? Closes #5010. ## Rationale for this change Native Parquet scans cannot currently apply Spark's legacy Julian/Gregorian rebasing safely before Parquet pruning, filters, and aggregates consume datetime values. This can produce incorrect results for files written by Spark 2.x or with legacy datetime or INT96 rebasing. I want to merge this conservative fallback first to restore correctness with a small, low-risk change. The native implementation in #5047 should remain a follow-up and can restore native acceleration after its broader per-file rebasing behavior is reviewed. ## What changes are included in this PR? - Inspect Parquet footer metadata and configured read modes before converting a scan to Comet. - Fall back to Spark when any input file may require legacy datetime or INT96 rebasing, or when metadata inspection fails. - Remove the dead datetime-rebase configuration and the unused `SparkParquetOptions.use_legacy_date_timestamp_or_ntz` field. - Update scan compatibility documentation and add regression coverage for the fallback. ## How are these changes tested? - `make core` - `git diff --check` - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadV1Suite fallback for Parquet datetime rebasing" -Dscalastyle.skip=true` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
