andygrove opened a new pull request, #3796: URL: https://github.com/apache/datafusion-comet/pull/3796
## Summary Fixes several issues with the `array_compact` expression found during an audit: - **Spark 4.0 support**: On Spark 4.0, `array_compact` rewrites to `KnownNotContainsNull(ArrayFilter(IsNotNull))`. Because `KnownNotContainsNull` was not handled by Comet, the expression always fell back to Spark on 4.0. This PR adds a case in the Spark 4.0 shim that unwraps `KnownNotContainsNull`, serializes the inner `ArrayFilter` using DataFusion's `array_remove_all`, and propagates the `containsNull=false` return type correctly. - **Return type bug**: `CometArrayCompact.convert` was hardcoding `ArrayType(elementType = elementType)` as the return type passed to DataFusion, which always set `containsNull=true`. Changed to use `expr.dataType` so the correct nullability metadata is emitted — matching Spark's behavior where `containsNull=false` after compaction on Spark 4.0. - **Incorrect `Incompatible` classification**: `CometArrayCompact.getSupportLevel` was returning `Incompatible(None)` with no explanation. The implementation is semantically correct (DataFusion's `array_remove_all(arr, null)` behaves the same as Spark's `ArrayFilter(IsNotNull)`), so this is changed to `Compatible()`. - **SQL test improvements**: Added `string`, `double`, and `array<array<int>>` (nested) type coverage; changed `spark_answer_only` to `query` to verify native execution. - **Remove Scala test skip**: The `assume(\!isSpark40Plus)` guard with a `// TODO fix for Spark 4.0.0` comment is removed now that Spark 4.0 is supported. - **Docs**: Updated `expressions.md` to mark `ArrayCompact` as supported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
