andygrove opened a new pull request, #4848: URL: https://github.com/apache/datafusion-comet/pull/4848
## Which issue does this PR close? Closes #4830. ## Rationale for this change `CometArrayFilter.convert` has a fast path that lowers `filter(arr, x -> x IS NOT NULL)` to the native `array_compact` serde, to avoid the per-batch JNI cost of the codegen dispatcher. The guard only checked that the lambda body is an `IsNotNull`, not that the `IsNotNull` operand is the lambda's element variable. As a result, a lambda whose body is `IsNotNull` of something other than the element (for example a captured column, `x -> c IS NOT NULL`) was incorrectly treated as `array_compact`. Spark keeps every element of `arr` when `c` is non-null and returns an empty array when `c` is null; the fast path instead dropped the null elements of `arr` regardless of `c`. This produced silently wrong results whenever `arr` contained null elements or the captured column was null. ## What changes are included in this PR? Tighten the guard in `CometArrayFilter.convert` to fire the `array_compact` fast path only when the `IsNotNull` operand is the lambda's own `NamedLambdaVariable` (matched by `exprId`, which also handles nested-lambda shadowing). Any other shape falls through to the codegen dispatcher, which runs Spark's own evaluation and matches Spark exactly. ## How are these changes tested? Added regression queries to `array_filter.sql`: a genuine `array_compact` case (`x -> x IS NOT NULL`) that must still fast-path, and the bug case (`x -> c IS NOT NULL` with a captured column and null array elements) that must match Spark. Verified the captured-column case fails against the pre-fix code and passes with the fix, and that the genuine fast path still runs natively. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
