[
https://issues.apache.org/jira/browse/SPARK-46912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021458#comment-18021458
]
Aparna Garg commented on SPARK-46912:
-------------------------------------
User 'MeltonSmith' has created a pull request for this issue:
https://github.com/apache/spark/pull/51314
> Spark-submit in cluster mode with standalone cluster uses wrong JAVA_HOME path
> ------------------------------------------------------------------------------
>
> Key: SPARK-46912
> URL: https://issues.apache.org/jira/browse/SPARK-46912
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit
> Affects Versions: 3.5.0
> Reporter: Danh Pham
> Priority: Major
> Labels: pull-request-available
>
> When run spark submit to a standalone cluster using cluster mode, the worker
> machine will use the JAVA_HOME value from remote machine instead of from
> worker machine.
> To reproduce:
> * Create a standalone cluster using docker compose, set JAVA_HOME in each
> worker different from local machine.
> * Run spark-submit, deploy-mode cluster
> * Monitor the log from worker, the driver will print out: DriverRunner:
> Launch Command: "<value from local JAVA_HOME>" "-cp" ...
> Reason:
> When Master create a new driver in receiveAndReply method, it uses the
> environment variables from submitter to build the driver description command.
> After that, when launch the driver, a new local (of worker) is built but it
> still use environment variable from driver description (which came from
> submitter). The result is the building java command will use the submitter
> java home path instead of worker path.
> Suggestion:
> Replace JAVA_HOME and SPARK_HOME in buildLocalCommand method ofÂ
> org.apache.spark.deploy.worker.CommandUtils by worker value
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
