andygrove commented on issue #5010: URL: https://github.com/apache/datafusion-comet/issues/5010#issuecomment-5152052847
Re-verified on `apache/main` @ `ba21f02de` (default Maven profile, Spark 4.1.2, macOS aarch64, debug build) while re-running the #4180 audit. Still reproduces, unchanged. Two things the original report did not pin down, both now measured: **All three read-mode values behave identically.** I tested `spark.sql.parquet.datetimeRebaseModeInRead` at `LEGACY`, `CORRECTED` and `EXCEPTION`, and also at the session default, against a file written by Spark 4.1 with `datetimeRebaseModeInWrite=LEGACY`. Comet returns `1000-01-06` in every one of the four, where Spark returns `1000-01-01`. The config is not merely mishandled for some values, it is never consulted. `spark.sql.parquet.int96RebaseModeInRead` behaves the same across the same four cases (`1000-01-01 12:34:56` reads back as `1000-01-06 12:41:58`). **The predicate case is silent, not just visibly shifted.** ```sql SELECT count(*) FROM legacy_dates WHERE d = date'1000-01-01' -- spark: 1 -- comet: 0 ``` **Carve-out:** part 1 of this issue, `spark.comet.exceptionOnDatetimeRebase` being dead code, is now tracked separately as #5195. It has a self-contained fix (wire the config up, or remove it) that does not depend on rebase support landing, and it was blocking this issue from reading as what it is. Part 2, the documentation scope, has since been addressed: `compatibility/scans.md` now describes the wider behaviour, names `datetimeRebaseModeInRead` and the `org.apache.spark.legacyDateTime` metadata, and links here. That leaves this issue as the underlying correctness gap only: either rebase legacy-calendar dates and timestamps in the native scan as Spark does, or fall back the affected scan to Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
