pedrumj2 opened a new pull request, #13513: URL: https://github.com/apache/gluten/pull/13513
## What changes are proposed in this pull request? - [UdfLoader::loadUdfLibrariesInternal](https://github.com/apache/gluten/blob/1503e13b5fdd953fa46e20a91ee90d71c0494b22/cpp/velox/udf/UdfLoader.cc#L59-L67) stores the handle `dlopen` returns even when it is null, and still logs `Successfully loaded udf library`. [UdfLoader::registerUdf](https://github.com/apache/gluten/blob/1503e13b5fdd953fa46e20a91ee90d71c0494b22/cpp/velox/udf/UdfLoader.cc#L153-L159) then looks up the library's `registerUdf` entry point through that null handle, so startup fails with `undefined symbol: registerUdf` in Gluten's own `libvelox.so`, and the real `dlopen` error is lost. - This PR throws a [GlutenException](https://github.com/apache/gluten/blob/1503e13b5fdd953fa46e20a91ee90d71c0494b22/cpp/core/utils/Exception.h#L57) with the `dlerror()` message as soon as `dlopen` fails, and stops caching the null handle. A library that fails to load still stops startup, as before. ## How was this patch tested? ### Local run Clone the repository: ```bash git clone https://github.com/apache/gluten.git && cd gluten ``` Build the Velox backend for Spark 3.5 in the CI image: ```bash docker run --rm -v "$PWD":/work/gluten -w /work/gluten apache/gluten:centos-9-jdk8 \ ./dev/buildbundle-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF --spark_version=3.5 ``` Start Spark with a file that is not a shared library as the UDF library: ```bash docker run --rm -v "$PWD":/work/gluten -w /work/gluten apache/gluten:centos-9-jdk8 bash -c ' echo "not a shared library" > /tmp/libbroken.so JAR=$(ls /work/gluten/package/target/gluten-velox-bundle-spark3.5_*.jar) cd /tmp /opt/shims/spark35/spark_home/bin/spark-sql --master "local[2]" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.driver.extraClassPath="$JAR" \ --conf spark.executor.extraClassPath="$JAR" \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///tmp/libbroken.so \ -e "SELECT 1"' ``` On `main` at [1503e13b5f](https://github.com/apache/gluten/commit/1503e13b5fdd953fa46e20a91ee90d71c0494b22), startup fails on a symbol missing from `libvelox.so`: ``` 26/11/01 01:27:26 ERROR NativeBackendInitializer: Failed to call native backend's initialize method org.apache.gluten.exception.GlutenException: Failed to load registerUdf in /tmp/gluten-b9d81a21-ce48-4d4c-98c6-15d1c7039cc0/jni/b7f5abff-fdf3-47ac-bde4-8cb580246c4a/gluten-7342595779360996524/libbroken.so: /tmp/gluten-b9d81a21-ce48-4d4c-98c6-15d1c7039cc0/jni/b7f5abff-fdf3-47ac-bde4-8cb580246c4a/gluten-7342595779360996524/linux/amd64/libvelox.so: undefined symbol: registerUdf … ``` Then check out this pull request's branch and repeat the build and run steps. Startup now fails with the loader's own error: ``` 26/11/01 01:31:32 ERROR NativeBackendInitializer: Failed to call native backend's initialize method org.apache.gluten.exception.GlutenException: Failed to load udf library /tmp/gluten-e082becb-bff3-4d5e-ae17-152f68e524ac/jni/a89a712f-25d3-4979-9cb3-72fed090ec60/gluten-5414739165593134846/libbroken.so: /tmp/gluten-e082becb-bff3-4d5e-ae17-152f68e524ac/jni/a89a712f-25d3-4979-9cb3-72fed090ec60/gluten-5414739165593134846/libbroken.so: file too short … ``` ### Automated tests Added `invalidLibraryFailsToLoad` to [MyUdfTest](https://github.com/apache/gluten/blob/1503e13b5fdd953fa46e20a91ee90d71c0494b22/cpp/velox/tests/MyUdfTest.cc#L27), which loads a file that is not a shared library. Built with `BUILD_TESTS` and `BUILD_EXAMPLES` on: ```bash cd cpp/build/velox/tests && ./my_udf_test ``` On `main`: ``` [ RUN ] MyUdfTest.invalidLibraryFailsToLoad I20261101 01:22:55.189849 1 UdfLoader.cc:65] Successfully loaded udf library: /tmp/gluten_my_udf_test_invalid.so /work/gluten/cpp/velox/tests/MyUdfTest.cc:57: Failure Expected: (error.find("Failed to load udf library " + path.string())) != (std::string::npos), actual: 18446744073709551615 vs 18446744073709551615 [ FAILED ] MyUdfTest.invalidLibraryFailsToLoad (0 ms) … ``` With this change: ``` [ RUN ] MyUdfTest.hivestringstring [ OK ] MyUdfTest.hivestringstring (0 ms) [ RUN ] MyUdfTest.invalidLibraryFailsToLoad [ OK ] MyUdfTest.invalidLibraryFailsToLoad (0 ms) … [ PASSED ] 2 tests. ``` ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Co-authored with Claude Code (claude-opus-5-5) <!-- Thank you for submitting a pull request! Here are some tips: 1. For first-time contributors, please read our contributing guide: https://github.com/apache/gluten/blob/main/CONTRIBUTING.md 2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work. 3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the Velox or ClickHouse backend, respectively. 4. If the PR is not ready for review, please mark it as a draft. --> <!-- Thank you for submitting a pull request! Here are some tips: 1. For first-time contributors, please read our contributing guide: https://github.com/apache/gluten/blob/main/CONTRIBUTING.md 2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work. 3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the Velox or ClickHouse backend, respectively. 4. If the PR is not ready for review, please mark it as a draft. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
