akshaytayal opened a new pull request, #13441: URL: https://github.com/apache/gluten/pull/13441
## What changes are proposed in this pull request? - **Pass RESPECT NULLS of `collect_set` to Velox.** `CollectRewriteRule` keeps Spark's `ignoreNulls` flag on `VeloxCollectSet`, but `HashAggregateExecTransformer` only sent the aggregate's children to Velox, so `collect_set(x) RESPECT NULLS` ran as Velox `collect_set(x)` and silently dropped nulls. Velox's Spark `collect_set` already accepts the flag as a constant second argument (`collect_set(T, boolean)`); this passes `false` there for RESPECT NULLS, in both the aggregate measure and the function signature (`VeloxIntermediateData.getRawInputs`). IGNORE NULLS (the default, and the only mode before Spark 4.2) keeps the one-argument form, so other plans don't change. The Partial step uses the companion `collect_set_partial`, which receives the constant since facebookincubator/velox#19446 (already in `dft-2026_10_23`). - **`array_join with nullable nullReplacement under whole-stage codegen`** (new in Spark 4.2) asserts the query runs in `WholeStageCodegenExec`; Gluten offloads `array_join` to `ProjectExecTransformer`. The parent test is excluded and a Gluten version asserts `array_join` is offloaded and keeps the same `checkAnswer` checks with codegen on and off. Re-enables `collect_set with RESPECT NULLS preserves null in set` and the `array_join` test in the Spark 4.2 UTs. The other #13176 tests need Velox changes and stay disabled (#13438, #13439, #13440). Part of #13176 (umbrella #13181). ## How was this patch tested? Locally on Spark 4.2 with native libraries matching current main (Velox `dft-2026_10_23`): `GlutenDataFrameAggregateSuite` and `GlutenDataFrameFunctionsSuite`: 319 succeeded, 0 failed (before: 317 succeeded, 12 failed with all excludes removed). Probes covered global, grouped, mixed RESPECT/IGNORE NULLS, and fused/non-fused Partial+Final plans. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot CLI (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
