pedrumj2 opened a new pull request, #13600:
URL: https://github.com/apache/gluten/pull/13600

   ## What changes are proposed in this pull request?
   
   Native 
[transform](https://spark.apache.org/docs/3.5.6/api/sql/index.html#transform) 
and [filter](https://spark.apache.org/docs/3.5.6/api/sql/index.html#filter) now 
run lambdas that take the element index, such as `transform(arr, (x, i) -> x + 
i)`, instead of falling back to the JVM.
   
   
[genArrayTransformTransformer](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L269-L271)
 and 
[genArrayFilterTransformer](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L219-L221)
 throw for a lambda with two arguments. 
[facebookincubator/velox#16211](https://github.com/facebookincubator/velox/pull/16211)
 and 
[facebookincubator/velox#16334](https://github.com/facebookincubator/velox/pull/16334)
 added the index variants of Velox's Spark `transform` and `filter`, this 
repo's Velox has carried them since apache/gluten#11624 and 
apache/gluten#11717, and the 
[dft-2026_11_06](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/functions/sparksql/registration/RegisterArray.cpp#L255-L258)
 tag registers both. This P
 R removes the two throws.
   
   A projection that holds such a lambda now stays native, together with every 
other expression in it.
   
   ## How was this patch tested?
   
   ### Reproduction
   
   1. Clone the repo and check out the base commit:
   
   ```bash
   git clone https://github.com/apache/gluten.git
   cd gluten
   git checkout cdf5292884895126825e4349938f2da3959fc812
   ```
   
   2. Build the native libraries and the bundle jar:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     source /opt/rh/gcc-toolset-12/enable
     cd /work/gluten
     ./dev/builddeps-veloxbe.sh --run_setup_script=OFF
     ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
   '
   ```
   
   3. Fetch Spark 3.5.5:
   
   ```bash
   curl -sSL 
https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | 
tar -xz
   ```
   
   4. Write the input table and the query:
   
   ```bash
   cat > write_input.sql <<'EOF'
   CREATE TABLE t USING parquet AS SELECT array(0, 2, 3) AS arr;
   EOF
   cat > query.sql <<'EOF'
   EXPLAIN SELECT transform(arr, (x, i) -> x + i), filter(arr, (x, i) -> x > i) 
FROM t;
   SELECT transform(arr, (x, i) -> x + i), filter(arr, (x, i) -> x > i) FROM t;
   EOF
   ```
   
   5. Write the table with plain Spark, then query it with Gluten:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     cd /work/gluten
     rm -rf metastore_db spark-warehouse
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f 
write_input.sql 2>/dev/null
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
       --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
       --conf spark.plugins=org.apache.gluten.GlutenPlugin \
       --conf spark.memory.offHeap.enabled=true \
       --conf spark.memory.offHeap.size=4g \
       --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
       -f query.sql 2>err.log
     grep -m 1 "index argument" err.log
   '
   ```
   
   On the base commit, the projection runs on the JVM, and the grep finds 
Gluten's reason:
   
   ```
   == Physical Plan ==
   Project [transform(arr#5, lambdafunction((lambda x#6 + lambda i#7), lambda 
x#6, lambda i#7, false)) AS transform(arr, 
lambdafunction((namedlambdavariable() + namedlambdavariable()), 
namedlambdavariable(), namedlambdavariable()))#10, filter(arr#5, 
lambdafunction((lambda x#8 > lambda i#9), lambda x#8, lambda i#9, false)) AS 
filter(arr, lambdafunction((namedlambdavariable() > namedlambdavariable()), 
namedlambdavariable(), namedlambdavariable()))#11]
   +- VeloxColumnarToRow
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[arr#5] Batched: true, DataFilters: [], Format: Parquet, 
Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], 
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<arr:array<int>> 
NativeFilters: []
   
   
   [0,3,5]      [2,3]
    - Validation failed with exception from: ProjectExecTransformer, reason: 
transform on array with lambda using index argument is not supported yet
   ```
   
   On this pull request's branch, with the bundle jar rebuilt, step 5 returns 
the same result from a native plan, and the grep finds nothing:
   
   ```
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [transform(arr#5, lambdafunction((lambda x#6 
+ lambda i#7), lambda x#6, lambda i#7, false)) AS transform(arr, 
lambdafunction((namedlambdavariable() + namedlambdavariable()), 
namedlambdavariable(), namedlambdavariable()))#10, filter(arr#5, 
lambdafunction((lambda x#8 > lambda i#9), lambda x#8, lambda i#9, false)) AS 
filter(arr, lambdafunction((namedlambdavariable() > namedlambdavariable()), 
namedlambdavariable(), namedlambdavariable()))#11]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[arr#5] Batched: true, DataFilters: [], Format: Parquet, 
Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], 
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<arr:array<int>> 
NativeFilters: []
   
   
   [0,3,5]      [2,3]
   ```
   
   Spark without Gluten, which is step 5's second command without the `--jars`, 
`--driver-class-path` and `--conf` options, returns the same result:
   
   ```
   == Physical Plan ==
   Project [transform(arr#5, lambdafunction((lambda x#6 + lambda i#7), lambda 
x#6, lambda i#7, false)) AS transform(arr, 
lambdafunction((namedlambdavariable() + namedlambdavariable()), 
namedlambdavariable(), namedlambdavariable()))#10, filter(arr#5, 
lambdafunction((lambda x#8 > lambda i#9), lambda x#8, lambda i#9, false)) AS 
filter(arr, lambdafunction((namedlambdavariable() > namedlambdavariable()), 
namedlambdavariable(), namedlambdavariable()))#11]
   +- *(1) ColumnarToRow
      +- FileScan parquet spark_catalog.default.t[arr#5] Batched: true, 
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<arr:array<int>>
   
   
   [0,3,5]      [2,3]
   ```
   
   ### Automated tests
   
   The `filter` and `transform` tests in 
[ScalarFunctionsValidateSuite](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala#L1058-L1103)
 each gain an index lambda, over null elements and null and empty arrays, 
checked to run natively.
   
   With this pull request's tests on the base commit's code, both tests fail on 
the new cases:
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
"-Dsuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite 
@filter,org.apache.gluten.functions.ScalarFunctionsValidateSuite @transform"
   ```
   
   ```
   - filter *** FAILED ***
     executedPlan.exists(((plan: org.apache.spark.sql.execution.SparkPlan) => 
tag.runtimeClass.isInstance(plan))) was false Expect ProjectExecTransformer 
exists in executedPlan:
      Project [filter(value#144, lambdafunction((lambda i#183 < 
(size(value#144, true) - 1)), lambda x#182, lambda i#183, false)) AS res#181]
     +- VeloxColumnarToRow
   …
   - transform *** FAILED ***
     executedPlan.exists(((plan: org.apache.spark.sql.execution.SparkPlan) => 
tag.runtimeClass.isInstance(plan))) was false Expect ProjectExecTransformer 
exists in executedPlan:
      Project [transform(arr#332, lambdafunction((lambda x#351 + lambda i#352), 
lambda x#351, lambda i#352, false)) AS transform(arr, 
lambdafunction((namedlambdavariable() + namedlambdavariable()), 
namedlambdavariable(), namedlambdavariable()))#353]
     +- VeloxColumnarToRow
   …
   Tests: succeeded 0, failed 2, canceled 0, ignored 0, pending 0
   *** 2 TESTS FAILED ***
   ```
   
   With this change:
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite
   ```
   
   ```
   ScalarFunctionsValidateSuite:
   …
   - filter
   …
   - transform
   …
   Tests: succeeded 90, failed 0, canceled 0, ignored 0, pending 0
   All tests passed.
   ```
   
   ```bash
   ./build/mvn -o -B test -Pspark-ut -Pbackends-velox -Pspark-3.5 -pl 
gluten-ut/spark35 -Dspotless.check.skip=true -Dscalastyle.skip=true 
-Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.catalyst.expressions.GlutenHigherOrderFunctionsSuite,org.apache.spark.sql.GlutenDataFrameFunctionsSuite
   ```
   
   ```
   GlutenHigherOrderFunctionsSuite:
   …
   - ArrayTransform
   …
   - ArrayFilter
   …
   GlutenDataFrameFunctionsSuite:
   …
   - transform function - array for primitive type not containing null
   …
   - transform function - array for primitive type containing null
   …
   - transform function - array for non-primitive type
   …
   - filter function - index argument
   …
   Tests: succeeded 158, failed 0, canceled 0, ignored 4, pending 0
   All tests passed.
   ```
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Co-authored with Claude Code (claude-opus-5-5)
   
   <!--
   Thank you for submitting a pull request! Here are some tips:
   
   1. For first-time contributors, please read our contributing guide:
      https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
   2. If necessary, create a GitHub issue for discussion beforehand to avoid 
duplicate work.
   3. If the PR is specific to a single backend, include [VL] or [CH] in the PR 
title to indicate the
      Velox or ClickHouse backend, respectively.
   4. If the PR is not ready for review, please mark it as a draft.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to