andygrove opened a new pull request, #3889:
URL: https://github.com/apache/datafusion-comet/pull/3889
## Which issue does this PR close?
Improves test coverage and correctness of `array_compact` support level.
## Rationale for this change
`CometArrayCompact.getSupportLevel` returned `Incompatible(None)`, but this
was incorrect:
- On Spark 3.x, the existing tests pass without `allowIncompatible`,
confirming the implementation matches Spark behavior.
- On Spark 4.0, `ArrayCompact` is a `RuntimeReplaceable` that Spark rewrites
to `ArrayFilter(child, IsNotNull(lambda))` before it reaches Comet's serde, so
`CometArrayCompact` is never invoked. The `CometArrayFilter` serde already
handles this case as `Compatible()`.
The test coverage was also limited to integer arrays only.
## What changes are included in this PR?
- Change `CometArrayCompact.getSupportLevel` from `Incompatible(None)` to
`Compatible()`
- Expand `array_compact.sql` test coverage:
- Add string element type tests (via table with column references)
- Add double element type test
- Add nested array type test (removes null arrays from outer, preserves
null elements in inner)
- Add empty string preservation test (not confused with nulls)
- Upgrade existing queries from `spark_answer_only` to `query` mode to
assert native execution
- Remove unnecessary `ConfigMatrix` for dictionary encoding
## How are these changes tested?
SQL file tests in `array_compact.sql` run via `CometSqlFileTestSuite`. All
tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
