pedrumj2 opened a new pull request, #13584:
URL: https://github.com/apache/gluten/pull/13584

   ## What changes are proposed in this pull request?
   
   Native 
[arrays_overlap](https://spark.apache.org/docs/3.5.6/api/sql/index.html#arrays_overlap)
 now treats `-0.0` and `0.0` as different values, as Spark does, so it returns 
`false` for `array(-0.0D)` and `array(0.0D)` instead of `true`.
   
   
[facebookincubator/velox#19805](https://github.com/facebookincubator/velox/pull/19805)
 added Spark's `arrays_overlap` to Velox, and this repo's Velox has included it 
since apache/gluten#13540. 
[registerPrestoVectorFunctions](https://github.com/apache/gluten/blob/01e71ec06b814c7b63af0306c24d9974bd2a30e8/cpp/velox/operators/functions/RegistrationAllFunctions.cc#L41-L44)
 still registers Velox's Presto `arrays_overlap` under the same name, and 
native queries run that version, which treats `-0.0` and `0.0` as equal. That 
registration has been the only one in `registerPrestoVectorFunctions` since 
apache/gluten#13511, so this PR removes the function and its call. Native 
queries that compare arrays of floating point values now return Spark's answer 
on Spark 4.1 and earlier.
   
   
[NormalizeFloatingNumbers](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NormalizeFloatingNumbers.scala#L112)
 normalizes `arrays_overlap`'s inputs on Spark 4.2, so Spark 4.2 returns `true` 
here, but Gluten passes that normalization through without applying it and 
returns `false`. Until apache/gluten#13565 maps the normalization to Velox, 
this PR excludes Spark 4.2's [SPARK-54918: arrays_overlap normalizes -0.0 to 
+0.0](https://github.com/apache/spark/blob/v4.2.0/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala#L6534-L6539)
 test in 
[VeloxTestSettings](https://github.com/apache/gluten/blob/01e71ec06b814c7b63af0306c24d9974bd2a30e8/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala#L1025),
 and runs the `arrays_overlap with signed zeros` test it adds to 
[ScalarFunctionsValidateSuite](https://github.com/apache/gluten/blob/01e71ec06b814c7b63af0306c24d9974bd2a30e
 
8/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala#L1225)
 on Spark 4.1 and earlier. TODO comments mark both for removal once 
apache/gluten#13565 lands.
   
   ## How was this patch tested?
   
   ### Reproduction
   
   1. Clone the repo and check out the base commit:
   
   ```bash
   git clone https://github.com/apache/gluten.git
   cd gluten
   git checkout 01e71ec06b814c7b63af0306c24d9974bd2a30e8
   ```
   
   2. Build the native libraries and the bundle jar:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     source /opt/rh/gcc-toolset-12/enable
     cd /work/gluten
     ./dev/builddeps-veloxbe.sh --run_setup_script=OFF
     ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
   '
   ```
   
   3. Fetch Spark 3.5.5:
   
   ```bash
   curl -sSL 
https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | 
tar -xz
   ```
   
   4. Write the input table and the query:
   
   ```bash
   cat > write_input.sql <<'EOF'
   CREATE TABLE arrays USING parquet AS SELECT array(-0.0D) AS a, array(0.0D) 
AS b;
   EOF
   cat > query.sql <<'EOF'
   SELECT arrays_overlap(a, b) FROM arrays;
   EXPLAIN SELECT arrays_overlap(a, b) FROM arrays;
   EOF
   ```
   
   5. Write the table with plain Spark, then query it with Gluten:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     cd /work/gluten
     rm -rf metastore_db spark-warehouse
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f 
write_input.sql 2>/dev/null
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
       --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
       --conf spark.plugins=org.apache.gluten.GlutenPlugin \
       --conf spark.memory.offHeap.enabled=true \
       --conf spark.memory.offHeap.size=4g \
       --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
       -f query.sql 2>/dev/null
   '
   ```
   
   On the base commit, the native plan returns `true`:
   
   ```
   true
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [arrays_overlap(a#0, b#1) AS 
arrays_overlap(a, b)#10]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.arrays[a#0,b#1] Batched: true, DataFilters: [], Format: 
Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/arrays], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<a:array<double>,b:array<double>> 
NativeFilters: []
   ```
   
   On this pull request's branch, rerunning steps 2 and 5 returns `false`, 
still in the native plan:
   
   ```
   false
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [arrays_overlap(a#0, b#1) AS 
arrays_overlap(a, b)#10]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.arrays[a#0,b#1] Batched: true, DataFilters: [], Format: 
Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/arrays], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<a:array<double>,b:array<double>> 
NativeFilters: []
   ```
   
   Spark without Gluten, which is step 5's second command without the `--jars`, 
`--driver-class-path` and `--conf` options, returns the same `false`:
   
   ```
   false
   == Physical Plan ==
   *(1) Project [arrays_overlap(a#0, b#1) AS arrays_overlap(a, b)#10]
   +- *(1) ColumnarToRow
      +- FileScan parquet spark_catalog.default.arrays[a#0,b#1] Batched: true, 
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/arrays], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<a:array<double>,b:array<double>>
   ```
   
   ### Automated tests
   
   `arrays_overlap with signed zeros` compares Gluten with Spark on arrays that 
hold `-0.0` and `0.0`.
   
   On the base commit's native libraries, the new test fails:
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
-Dsuites="org.apache.gluten.functions.ScalarFunctionsValidateSuite 
arrays_overlap"
   ```
   
   ```
   - arrays_overlap
   - arrays_overlap with signed zeros *** FAILED ***
   …
     !== Correct Answer - 2 ==   == Gluten Answer - 2 ==
      struct<>                   struct<>
     ![false]                    [true]
      [true]                     [true] (GlutenQueryTest.scala:460)
   …
   Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
   ```
   
   With this change:
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite
   ```
   
   ```
   ScalarFunctionsValidateSuite:
   …
   - arrays_overlap
   …
   - arrays_overlap with signed zeros
   …
   Tests: succeeded 91, failed 0, canceled 0, ignored 0, pending 0
   All tests passed.
   ```
   
   ```bash
   ./build/mvn -o -B test -Pspark-ut -Pbackends-velox -Pspark-3.5 -pl 
gluten-ut/spark35 -Dspotless.check.skip=true -Dscalastyle.skip=true 
-Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.catalyst.expressions.GlutenCollectionExpressionsSuite,org.apache.spark.sql.GlutenDataFrameFunctionsSuite
   ```
   
   ```
   GlutenCollectionExpressionsSuite:
   …
   - ArraysOverlap
   …
   GlutenDataFrameFunctionsSuite:
   …
   - arrays_overlap function
   …
   Tests: succeeded 194, failed 0, canceled 0, ignored 7, pending 0
   All tests passed.
   ```
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Co-authored with Claude Code (claude-opus-5-5)
   
   <!--
   Thank you for submitting a pull request! Here are some tips:
   
   1. For first-time contributors, please read our contributing guide:
      https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
   2. If necessary, create a GitHub issue for discussion beforehand to avoid 
duplicate work.
   3. If the PR is specific to a single backend, include [VL] or [CH] in the PR 
title to indicate the
      Velox or ClickHouse backend, respectively.
   4. If the PR is not ready for review, please mark it as a draft.
   -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to