peterxcli opened a new pull request, #5047: URL: https://github.com/apache/datafusion-comet/pull/5047
## Which issue does this PR close? Closes #5010. ## Rationale for this change Native Parquet scans did not apply Spark's legacy Julian/Gregorian datetime rebasing. Files written by Spark 2.x or with a `LEGACY` write mode could therefore return shifted ancient `DATE` and `TIMESTAMP` values. Rebasing only final output is insufficient because Parquet predicates and native filters or aggregates can consume the encoded values first. Spark resolves rebasing separately for every file, with footer metadata taking precedence over reader configuration. Comet needs the same behavior to keep native scans correct for legacy, corrected, and mixed-file inputs. ## What changes are included in this PR? - Propagate the Parquet datetime and INT96 read modes plus the executor JVM timezone to native scans. - Resolve the effective rebase specification from each file's Spark version, legacy markers, and timezone metadata. - Rebase `DATE`, `TIMESTAMP_MILLIS`, `TIMESTAMP_MICROS`, and INT96 values before native filtering and aggregation. - Use Spark's Julian-to-Gregorian lookup table, with Spark JNI fallback for table misses and BCE timestamps. - Raise Spark's normal rebase exception in `EXCEPTION` mode. - Remove the dead datetime-rebase configuration and obsolete native scan limitation. - Add focused Rust tests and Spark-derived Parquet tests for mixed per-file modes, configuration precedence, and error propagation. ## How are these changes tested? - `make core` - `cargo fmt --all -- --check` - `cargo clippy --all-targets --workspace -- -D warnings` - `cargo test -p datafusion-comet datetime_rebase --no-default-features` - `./mvnw test -Dtest=none '-Dsuites=org.apache.comet.parquet.ParquetReadV1Suite native scan rebases legacy' -Dscalastyle.skip=true` - `./mvnw test -Dtest=none '-Dsuites=org.apache.comet.parquet.ParquetReadV1Suite native scan honors per-file datetime rebase modes' -Dscalastyle.skip=true` - `./mvnw test -Dtest=none '-Dsuites=org.apache.spark.sql.comet.PlanDataInjectorSuite native scan injector records the executor JVM timezone' -Dscalastyle.skip=true` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
