pedrumj2 opened a new pull request, #13274: URL: https://github.com/apache/gluten/pull/13274
## What changes are proposed in this pull request? A UDF loaded through [`udfLibraryPaths`](https://github.com/apache/gluten/blob/main/docs/developers/VeloxUDF.md) now keeps the field names of every struct in its signature, so a named struct argument binds and the fields of a returned struct can be selected by name. apache/gluten#13016 let such a UDF be called by its own name for apache/gluten#13014, behind [`spark.gluten.sql.columnar.backend.velox.nativeUDF.bypassRegistration`](https://github.com/apache/gluten/blob/main/docs/velox-configuration.md). Today the above implementation does not support NameStructs. This diff aims to address that ## How was this patch tested? ### Reproduction Everything runs in the `apache/gluten:centos-9-jdk8` image. Run each block from the same empty directory. The queries call two example UDFs that this pull request adds to [`MyUDF.cc`](https://github.com/apache/gluten/blob/main/cpp/velox/udf/examples/MyUDF.cc): one takes a `struct<a:bigint,b:bigint>` and the other returns one. #### [step1] Get the source and Spark ```bash git clone https://github.com/apache/gluten.git curl -LO https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz tar -xzf spark-3.5.5-bin-hadoop3.tgz ``` Then check out this pull request's branch in `gluten` as a local branch named `pr`. #### [step2] Write the queries ```bash cat > repro.sql <<'EOF' SELECT myudf_sum_row(named_struct('a', a, 'b', b)) AS sum_ab FROM VALUES (1L, 2L), (3L, 4L) AS t(a, b); SELECT myudf_make_row(id).b AS b FROM VALUES (1L), (2L) AS t(id); EXPLAIN SELECT myudf_make_row(id).b AS b FROM VALUES (1L) AS t(id); EOF ``` #### [step3] Build Gluten with the UDF examples ```bash docker run --rm -v "$PWD":/work:z -w /work/gluten apache/gluten:centos-9-jdk8 bash -c ' source /opt/rh/gcc-toolset-12/enable export VELOX_BUILD_SHARED=ON ./dev/builddeps-veloxbe.sh --build_examples=ON --build_arrow=OFF --run_setup_script=OFF ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests' ``` #### [step4] Run the queries ```bash docker run --rm -v "$PWD":/work:z -w /work apache/gluten:centos-9-jdk8 \ spark-3.5.5-bin-hadoop3/bin/spark-sql --master 'local[2]' \ --jars gluten/package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.8.0-SNAPSHOT.jar \ --driver-class-path gluten/package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.8.0-SNAPSHOT.jar \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///work/gluten/cpp/build/velox/udf/examples/libmyudf.so \ --conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///work/gluten/cpp/build/velox/udf/examples/libmyudf.so \ --conf spark.gluten.sql.columnar.backend.velox.nativeUDF.bypassRegistration=true \ --hiveconf hive.cli.errors.ignore=true \ -f repro.sql ``` #### Before: the base commit with only the example UDFs from this pull request ```bash git -C gluten checkout --detach pr~1 git -C gluten checkout pr -- cpp/velox/udf/examples/MyUDF.cc ``` Then [step3] and [step4]: ``` 26/10/13 19:44:41 ERROR SparkSQLDriver: Failed in [SELECT myudf_sum_row(named_struct('a', a, 'b', b)) AS sum_ab FROM VALUES (1L, 2L), (3L, 4L) AS t(a, b)] org.apache.gluten.exception.GlutenNotSupportException: UDF myudf_sum_row -> struct<a:bigint,b:bigint> is not registered. at org.apache.spark.sql.expression.UDFResolver$.getUdfExpression(UDFResolver.scala:407) at org.apache.spark.sql.expression.UDFResolver$.$anonfun$getFunctionDescriptions$10(UDFResolver.scala:385) ... [FIELD_NOT_FOUND] No such struct field `b` in ``, ``. ... == Physical Plan == org.apache.spark.sql.AnalysisException: [FIELD_NOT_FOUND] No such struct field `b` in ``, ``. ``` #### After: this pull request ```bash git -C gluten checkout pr ``` Then [step3] and [step4]: ``` 3 7 2 3 == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [myudf_make_row(myudf_make_row, myudf_make_row, StructField(a,LongType,true), StructField(b,LongType,true), true, id#13L).b AS b#7L] +- ^(1) InputIteratorTransformer[id#13L] +- RowToVeloxColumnar +- LocalTableScan [id#13L] ``` ### Tests New cases cover a named struct argument and return value end to end in [`VeloxUdfSuite`](https://github.com/apache/gluten/blob/main/backends-velox/src/test/scala/org/apache/gluten/expression/VeloxUdfSuite.scala), decoding with and without names in [`ConverterUtilsSuite`](https://github.com/apache/gluten/blob/main/gluten-substrait/src/test/scala/org/apache/gluten/expression/ConverterUtilsSuite.scala), and nested structs, including structs inside an array and a map, in [`VeloxToSubstraitTypeTest`](https://github.com/apache/gluten/blob/main/cpp/velox/tests/VeloxToSubstraitTypeTest.cc). ``` ./build/mvn install -Pbackends-velox -Pspark-3.5 -Pfast-build -pl gluten-substrait,backends-velox -Dtest=none -Dsurefire.failIfNoSpecifiedTests=false -DwildcardSuites=org.apache.gluten.expression.ConverterUtilsSuite,org.apache.gluten.expression.VeloxUdfSuiteLocal -DtagsToExclude=org.apache.spark.tags.SkipTest ``` ``` - TimestampType emits PrecisionTimestampTZ(precision=6) and round-trips - TimestampNTZType emits PrecisionTimestamp(precision=6) and round-trips - column names for an attribute with a null name - struct field names are kept when the producer sets them - struct fields stay unnamed when the producer sets no names Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0 All tests passed. - test native hive udf - test native hive udaf - test native hive udaf in window - test udf fallback in partition filter - native udf with a plain name is callable without a hive udf class - native udf with a plain name fails at analysis when gluten is disabled - native udf with a named row argument is callable - native udf returning a named row exposes its field names Tests: succeeded 8, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` ``` cpp/build/velox/tests/velox_plan_conversion_test ``` ``` [ OK ] VeloxToSubstraitTypeTest.rowFieldNames (0 ms) [ OK ] VeloxToSubstraitTypeTest.rowFieldNamesInContainers (0 ms) [==========] 57 tests from 9 test suites ran. (644 ms total) [ PASSED ] 50 tests. [ SKIPPED ] 7 tests, listed below: [ SKIPPED ] VeloxSubstraitRoundTripTest.sum [ SKIPPED ] VeloxSubstraitRoundTripTest.sumAndCount [ SKIPPED ] VeloxSubstraitRoundTripTest.sumGlobal [ SKIPPED ] VeloxSubstraitRoundTripTest.sumMask [ SKIPPED ] VeloxSubstraitRoundTripTest.projectAs [ SKIPPED ] VeloxSubstraitRoundTripTest.avg [ SKIPPED ] VeloxSubstraitRoundTripTest.subField YOU HAVE 1 DISABLED TEST ``` ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Co-authored with Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
