kevinwilfong opened a new pull request, #13530:
URL: https://github.com/apache/gluten/pull/13530
Run cluster executors with the test JVM classpath instead of the shaded
package jar. The driver is already running unshaded module classes, so
spark.driver.extraClassPath cannot make it match the package jar. Remove the
unused gluten.package.jar property and its documentation.
Build driver UDF library URIs from absolute paths so the default relative
cpp.build.dir works. Enable by-name registration for the shared UDF tests.
Select the cluster suite with UDFTest and run the local and cluster suites
in separate JVMs in the dedicated CI job. Install the required modules once
before selecting each suite in backends-velox. Trigger that job when the UDF
suite or its workflow changes, as well as for native changes.
Validated against upstream d64a51dfd with rebuilt pinned native libraries,
Spark 3.5.5, Scala 2.12 and JDK 17: local 8/8, cluster 8/8, none skipped. The
CI profiles, tags and default relative library paths were used. Formatting,
license headers, workflow syntax and command checks passed. Linux GitHub
Actions execution is pending.
Originally authored with Claude Code; updated and validated with Codex.
Generated-by: Codex CLI 0.164.0
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
Fix and enable VeloxUdfSuiteCluster in the dedicated UDF CI job.
- Use the test JVM’s classpath for executors instead of the shaded Gluten
package jar. This prevents driver/executor class mismatches that cause UDAF
serialization failures and removes the gluten.package.jar requirement.
- Construct absolute file URIs for driver UDF libraries so the default
relative build paths work.
- Enable by-name UDF registration, matching the local suite’s configuration.
- Replace SkipTest with UDFTest and run the local and cluster suites in
separate JVMs to isolate registration state.
- Install required modules before the individual suite runs and trigger the
UDF job when the suite or workflow changes
## How was this patch tested?
Validated locally with rebuilt native libraries, Spark 3.5.5, Scala 2.12,
and JDK 17: both suites passed all eight tests, with none skipped.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GPT-6.1 Sol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
