This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f5227a64c2f3 [SPARK-57945][K8S] Avoid persisting Java options to disk
in `entrypoint.sh`
f5227a64c2f3 is described below
commit f5227a64c2f3b9a6e53098b9d1a2ae425fa3f341
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Jul 5 20:21:47 2026 -0700
[SPARK-57945][K8S] Avoid persisting Java options to disk in `entrypoint.sh`
### What changes were proposed in this pull request?
This PR removes the temporary file `java_opts.txt` from the K8s container
`entrypoint.sh` by reading `SPARK_JAVA_OPT_*` environment variables directly
into the array via process substitution, which works in both `bash` and `zsh`.
### Why are the changes needed?
The `java_opts.txt` file was introduced at Apache Spark 2.3.0.
- https://github.com/apache/spark/pull/20192
`SPARK_JAVA_OPT_*` carries `spark.executor.extraJavaOptions` verbatim,
which may contain sensitive values (e.g., keystore passwords). The file was
written in plaintext to the working directory and never deleted, so it could
outlive the process on persistent volumes (e.g., PV/hostPath).
Note that the scope of this PR is only preventing a persistent files. The
pods still have `env` variables .
### Does this PR introduce _any_ user-facing change?
No, the resulting Java options are identical.
### How was this patch tested?
Manually verified in both `bash` and `zsh` that options with spaces, the
empty case under `set -e`, and the numeric ordering are preserved.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57019 from dongjoon-hyun/SPARK-57945.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
index f9561b9aa4ed..a2ae7e356694 100755
---
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
+++
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
@@ -41,12 +41,10 @@ if [ -z "$JAVA_HOME" ]; then
fi
SPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*"
-env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' >
java_opts.txt
-if [ "$(command -v readarray)" ]; then
- readarray -t SPARK_EXECUTOR_JAVA_OPTS < java_opts.txt
-else
- SPARK_EXECUTOR_JAVA_OPTS=("${(@f)$(< java_opts.txt)}")
-fi
+SPARK_EXECUTOR_JAVA_OPTS=()
+while IFS= read -r opt; do
+ SPARK_EXECUTOR_JAVA_OPTS+=("$opt")
+done < <(env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed
's/[^=]*=\(.*\)/\1/g')
if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then
SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
