dongjoon-hyun opened a new pull request, #59088: URL: https://github.com/apache/spark/pull/59088
### What changes were proposed in this pull request? This PR is a backport of #59070 to `branch-4.0`. This PR aims to support `spark.executor.extraLibraryPath` for K8s executors. - `BasicExecutorFeatureStep` passes it to the executor container as `SPARK_EXECUTOR_LIBRARY_PATH`. - `entrypoint.sh` prepends `SPARK_EXECUTOR_LIBRARY_PATH` to `LD_LIBRARY_PATH` before launching the executor JVM. The previous cherry-pick to `branch-4.0` was reverted because the new unit test failed to compile. Unlike `master` and `branch-4.x`, `BasicExecutorFeatureStep` of `branch-4.0` still takes `SecurityManager` because SPARK-59479 (#58768) is not in `branch-4.0`. So, the new unit test passes `new SecurityManager(baseConf)` like the other test cases of `BasicExecutorFeatureStepSuite`. This is the only difference from the original commit. ### Why are the changes needed? Currently, only Standalone and YARN apply `spark.executor.extraLibraryPath`, and K8s silently ignores it. Like Standalone and YARN, this PR prepends the paths to the existing `LD_LIBRARY_PATH` so that the value from the image or `spark.executorEnv.LD_LIBRARY_PATH` is preserved. In addition, `apache/spark-docker` already supports `SPARK_EXECUTOR_LIBRARY_PATH` in `entrypoint.sh.template` via SPARK-59799 (apache/spark-docker#134). Since `add-dockerfiles.sh` generates `entrypoint.sh` of new Spark versions from this template, this PR makes `branch-4.0` consistent with its upcoming official docker images. ### Does this PR introduce _any_ user-facing change? Yes. `spark.executor.extraLibraryPath` takes effect for K8s executors when the image uses Spark's `entrypoint.sh`. Previously, it was silently ignored. ### How was this patch tested? Pass the CIs with the newly added unit test and K8s integration test. In addition, the new unit test and the compilation of K8s integration tests are verified locally. ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests 'kubernetes/testOnly *BasicExecutorFeatureStepSuite' 'kubernetes-integration-tests/Test/compile' ... [info] - SPARK-59798: extra library path gets translated into an environment variable (11 milliseconds) ... [info] Tests: succeeded 40, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
