andygrove opened a new pull request, #7103:
URL: https://github.com/apache/datafusion-comet/pull/7103
## Which issue does this PR close?
No issue. This fixes a test that has been failing on Spark 3.4 and 3.5 since
#4974 landed.
## Rationale for this change
`CometWindowExecSuite` "window: largest RANGE offset on DATE ORDER BY that
cannot wrap runs natively" (added in #4974) writes `0001-01-01` to Parquet. On
Spark 3.x `spark.sql.parquet.datetimeRebaseModeInWrite` defaults to
`EXCEPTION`, so the write throws
`INCONSISTENT_BEHAVIOR_CROSS_VERSION.WRITE_ANCIENT_DATETIME`. Spark 4.x
defaults to `CORRECTED`, so the test passes there.
The pull request tier and the merge queue only run Spark 4.1, so the break
was not seen when #4974 merged. It fails the `Spark 3.4` and `Spark 3.5`
`[exec]` jobs on any pull request that runs the other profiles (for example
#7096), and in the nightly.
## What changes are included in this PR?
The test now writes its fixture inside
`withSQLConf("spark.sql.parquet.datetimeRebaseModeInWrite" -> "CORRECTED")`.
Nothing else changes, and the test still reads the file under Spark's default
read mode.
## How are these changes tested?
Ran the test locally on the Spark 3.5 profile with a debug native build. It
failed before the change at the write and passes after (`Tests: succeeded 1,
failed 0`). I did not run it on 3.4, where the write config and default are the
same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
