andygrove opened a new pull request, #4370:
URL: https://github.com/apache/datafusion-comet/pull/4370
## Which issue does this PR close?
Closes #.
Builds on the idea behind #4368.
## Rationale for this change
`CometTestBase` was pinning `spark.sql.ansi.enabled` to `false`, so the
Spark 4.0/4.1/4.2 test profiles never actually exercised the ANSI behavior real
Spark 4 users get out of the box. Tests that pass against Comet today may
regress in production because we never run them with the same defaults.
## What changes are included in this PR?
- Stop setting `ANSI_ENABLED` in `CometTestBase.sparkConf`. Spark 3.x picks
up its own default (off), Spark 4.x picks up its own default (on).
- For suites whose tests predominantly exercise silent-overflow /
wrap-around / null-on-bad-input semantics, override `sparkConf` at the suite
level to disable ANSI (tests that target ANSI behavior already opt in via
`withSQLConf(ANSI_ENABLED -> "true")`):
- `CometFuzzMathSuite`
- `CometCastSuite`
- `CometAggregateSuite`
- `CometSqlFileTestSuite`: prepend `ANSI_ENABLED -> "false"` to the conf
list passed into `withSQLConf`. The `_ansi.sql` fixtures that include
`--CONFIG: spark.sql.ansi.enabled=true` still win, because their values are
applied later.
- Per-test `withSQLConf(SQLConf.ANSI_ENABLED.key -> "false")` wraps for the
long tail in `CometExpressionSuite`, `CometArrayExpressionSuite`,
`CometMathExpressionSuite`, `CometDateTimeUtilsSuite`, `CometExecSuite`, and
`ParquetReadSuite`.
## How are these changes tested?
The existing tests are the test. Ran the affected suites under `-Pspark-4.0`
locally (892 tests across 10 suites, 0 failures):
| Suite | Tests | Result |
|---|---|---|
| `CometFuzzMathSuite` | 30 | pass |
| `CometCastSuite` | 165 | pass |
| `CometAggregateSuite` | 81 | pass |
| `CometExpressionSuite` | 127 | pass |
| `CometArrayExpressionSuite` | 39 | pass |
| `CometMathExpressionSuite` | 7 | pass |
| `CometDateTimeUtilsSuite` | 8 | pass |
| `CometExecSuite` | 124 | pass |
| `CometSqlFileTestSuite` | 258 | pass |
| `ParquetReadV1Suite` | 53 | pass |
CI will sweep Spark 4.0/4.1/4.2 across all the partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
