[
https://issues.apache.org/jira/browse/SPARK-46912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012359#comment-18012359
]
Melton commented on SPARK-46912:
--------------------------------
Anyway, env should be set per worker in SA cluster.
I don't see a case when JAVA_HOME from submitter machine should be used (only
if you, let's say, have a few java versions installed on a worker, and one
wants to make a "hint" for which java to use. But, i think, it's quite
unrealistic scenario)
> Spark-submit in cluster mode with standalone cluster uses wrong JAVA_HOME path
> ------------------------------------------------------------------------------
>
> Key: SPARK-46912
> URL: https://issues.apache.org/jira/browse/SPARK-46912
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit
> Affects Versions: 3.5.0
> Reporter: Danh Pham
> Priority: Major
> Labels: pull-request-available
>
> When run spark submit to a standalone cluster using cluster mode, the worker
> machine will use the JAVA_HOME value from remote machine instead of from
> worker machine.
> To reproduce:
> * Create a standalone cluster using docker compose, set JAVA_HOME in each
> worker different from local machine.
> * Run spark-submit, deploy-mode cluster
> * Monitor the log from worker, the driver will print out: DriverRunner:
> Launch Command: "<value from local JAVA_HOME>" "-cp" ...
> Reason:
> When Master create a new driver in receiveAndReply method, it uses the
> environment variables from submitter to build the driver description command.
> After that, when launch the driver, a new local (of worker) is built but it
> still use environment variable from driver description (which came from
> submitter). The result is the building java command will use the submitter
> java home path instead of worker path.
> Suggestion:
> Replace JAVA_HOME and SPARK_HOME in buildLocalCommand method ofÂ
> org.apache.spark.deploy.worker.CommandUtils by worker value
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
