pedrumj2 opened a new pull request, #13604: URL: https://github.com/apache/gluten/pull/13604
## What changes are proposed in this pull request? When ANSI mode is on, Spark's [try_add](https://spark.apache.org/docs/3.5.6/api/sql/index.html#try_add), [try_subtract](https://spark.apache.org/docs/3.5.6/api/sql/index.html#try_subtract) and [try_multiply](https://spark.apache.org/docs/3.5.6/api/sql/index.html#try_multiply) on integers throw an error when computing one of their inputs fails. This PR makes the native path follow the same pattern: | Query with ANSI mode on, where `z = 0` | Native before this PR | Native with this PR | Spark | |---|---|---|---| | `try_add(CAST(a / z AS INT), 1)` | `NULL` | throws division by zero | throws `DIVIDE_BY_ZERO` | The native path now sends Velox a different function for each version of try: | Spark function | Sent to Velox before this PR | Sent to Velox with this PR | |---|---|---| | `try_add(left, right)` | `try(checked_add(left, right))` | `try_add(left, right)` | | `try_subtract(left, right)` | `try(checked_subtract(left, right))` | `try_subtract(left, right)` | | `try_multiply(left, right)` | `try(checked_multiply(left, right))` | `try_multiply(left, right)` | Velox's [`try`](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/expression/TryExpr.cpp#L21-L48) turns every error inside it into `NULL`. [facebookincubator/velox#19463](https://github.com/facebookincubator/velox/pull/19463) added Velox's own `try_add`, `try_subtract` and `try_multiply`, which return `NULL` only when the operation overflows. This repo's Velox, the [dft-2026_11_06](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/functions/sparksql/registration/RegisterMath.cpp#L176-L178) tag, has these operators. ## How was this patch tested? ### Reproduction 1. Clone the repo and check out the base commit: ```bash git clone https://github.com/apache/gluten.git cd gluten git checkout cdf5292884895126825e4349938f2da3959fc812 ``` 2. Build the native libraries and the bundle jar: ```bash docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' source /opt/rh/gcc-toolset-12/enable cd /work/gluten ./dev/builddeps-veloxbe.sh --run_setup_script=OFF ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests ' ``` 3. Fetch Spark 3.5.5: ```bash curl -sSL https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | tar -xz ``` 4. Write the input table and the query: ```bash cat > write_input.sql <<'EOF' CREATE TABLE nums USING parquet AS SELECT 1 AS a, 0 AS z; EOF cat > query.sql <<'EOF' EXPLAIN SELECT try_add(CAST(a / z AS INT), 1) FROM nums; SELECT try_add(CAST(a / z AS INT), 1) FROM nums; EOF ``` 5. Write the table with plain Spark, then query it with Gluten, with ANSI mode on and Gluten's ANSI fallback off: ```bash docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' cd /work/gluten rm -rf metastore_db spark-warehouse spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f write_input.sql 2>/dev/null BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar) spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \ --jars "$BUNDLE" --driver-class-path "$BUNDLE" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.sql.ansi.enabled=true \ --conf spark.gluten.sql.ansiFallback.enabled=false \ -f query.sql 2>err.log grep -m 1 -E "DIVIDE_BY_ZERO|division by zero" err.log ' ``` On the base commit, the query runs natively and returns `NULL`. The grep finds no error: ``` == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [(cast((cast(a#5 as double) / cast(z#6 as double)) as int) + 1) AS try_add(CAST((a / z) AS INT), 1)#7] +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.nums[a#5,z#6] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/nums], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:int,z:int> NativeFilters: [] NULL ``` On this pull request's branch, rerunning steps 2 and 5 shows the same native plan. The query now fails with Velox's error: ``` == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [(cast((cast(a#5 as double) / cast(z#6 as double)) as int) + 1) AS try_add(CAST((a / z) AS INT), 1)#7] +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.nums[a#5,z#6] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/nums], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:int,z:int> NativeFilters: [] Reason: Reason: (0 vs. 0) division by zero ``` Spark without Gluten fails the same way. To run it, remove every option from step 5's second command except `--master`, [`--conf spark.sql.ansi.enabled=true`](https://spark.apache.org/docs/3.5.6/sql-ref-ansi-compliance.html) and `-f query.sql`: ``` == Physical Plan == *(1) Project [(cast((cast(a#5 as double) / cast(z#6 as double)) as int) + 1) AS try_add(CAST((a / z) AS INT), 1)#7] +- *(1) ColumnarToRow +- FileScan parquet spark_catalog.default.nums[a#5,z#6] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/nums], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:int,z:int> org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. ``` ### Automated tests `integral try_add, try_subtract and try_multiply` in [ArithmeticAnsiValidateSuite](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala) checks on Velox that an overflow returns `NULL` and that an input that divides by zero throws. On the base commit, the new test fails: ```bash ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -Pspark-ut -pl backends-velox -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.gluten.functions.ArithmeticAnsiValidateSuite -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true ``` ``` - integral try_add, try_subtract and try_multiply *** FAILED *** Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (ArithmeticAnsiValidateSuite.scala:204) … Tests: succeeded 12, failed 1, canceled 0, ignored 0, pending 0 ``` With this change, the same command passes: ``` … - integral try_add, try_subtract and try_multiply … Tests: succeeded 13, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` ```bash ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -Pspark-ut -pl backends-velox -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.gluten.functions.MathFunctionsValidateSuite -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true ``` ``` MathFunctionsValidateSuiteAnsiOn: … - try_add … - try_divide … - try_multiply … - try_subtract … MathFunctionsValidateSuite: … - try_add … - try_divide … - try_multiply … - try_subtract … Tests: succeeded 51, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` ```bash ./build/mvn -o -B test -Pspark-ut -Pbackends-velox -Pspark-3.5 -pl gluten-ut/spark35 -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.spark.sql.catalyst.expressions.GlutenTryEvalSuite,org.apache.spark.sql.catalyst.expressions.GlutenArithmeticExpressionSuite,org.apache.spark.sql.GlutenMathFunctionsSuite -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true ``` ``` GlutenTryEvalSuite: … GlutenArithmeticExpressionSuite: … GlutenMathFunctionsSuite: … Tests: succeeded 99, failed 0, canceled 0, ignored 1, pending 0 All tests passed. ``` ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Co-authored with Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
