rdtr opened a new pull request, #13565: URL: https://github.com/apache/gluten/pull/13565
## What changes are proposed in this pull request? Since Spark 4.2 (SPARK-54918), `NormalizeFloatingNumbers` rewrites `array_distinct(a)` to `array_distinct(knownfloatingpointnormalized(transform(a, x -> normalizenanandzero(x))))`, and does the same for `array_union`, `array_intersect`, `array_except` and `arrays_overlap`. Gluten offloads that plan, but `ExpressionConverter` converted `NormalizeNaNAndZero` as a pass-through (`ChildTransformer`), so the normalization was dropped. For `array_distinct(array(-0.0, 0.0))` Velox returned `[-0.0]` where Spark returns `[0.0]`. facebookincubator/velox#19835 added the Spark function `normalize_nan_and_zero`, which is in the pinned Velox since `dft-2026_11_05`. This adds `SparkPlanExecApi.genNormalizeNaNAndZeroTransformer`. It keeps the pass-through by default, so the ClickHouse and Bolt backends don't change, and `VeloxSparkPlanExecApi` maps `NormalizeNaNAndZero` to `normalize_nan_and_zero`. The Substrait name in `ExpressionNames` changes from `normalize_nanand_zero` to `normalize_nan_and_zero`. Nothing else used it. `KnownFloatingPointNormalized` stays a pass-through. Spark also adds `NormalizeNaNAndZero` to floating-point join keys, grouping keys and window partition keys. Those now run the Velox function too, which matches Spark's semantics. This re-enables the 7 SPARK-54918 tests in `GlutenDataFrameFunctionsSuite` on Spark 4.2. `collect_set` (SPARK-57298, #13440) normalizes inside the aggregate in Spark and needs a separate change. Fixes #13439 ## How was this patch tested? Ran `GlutenDataFrameFunctionsSuite`, `GlutenDataFrameAggregateSuite` and `GlutenJoinSuite` with `SPARK_TESTING=true`, `SPARK_ANSI_SQL_MODE=false`, `-Pdelta` and `-Dspark.test.home` set, as in the CI jobs, against a native build of the pinned Velox (`dft-2026_11_05`): - Spark 4.2: 366 succeeded, 0 failed, including all 11 SPARK-54918 tests. - Spark 3.5: 267 succeeded, 0 failed. On main the 7 re-enabled tests fail with `isPositiveZero(r.head) was false`. With this change, `array_distinct` over a column of `array<double>` runs as `ProjectExecTransformer [array_distinct(knownfloatingpointnormalized(transform(value, lambdafunction(knownfloatingpointnormalized(normalizenanandzero(lambda arg)), ...))))]`, so the normalization runs in Velox. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
