[
https://issues.apache.org/jira/browse/SPARK-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-3734:
------------------------------
Description:
If you use {{spark-submit}} in {{cluster}} mode to submit a job to a Spark
Standalone cluster, and the {{JAVA_HOME}} environment variable is set on the
submitting machine, then DriverRunner will attempt to use the _submitter's_
JAVA_HOME to launch the driver process (instead of the worker's JAVA_HOME),
which can cause the job to fail unless the submitter and worker have Java
installed in the same location.
This has a pretty simple fix: read JAVA_HOME from {{sys.env}} instead of
{{command.environment}}; PR pending shortly.
was:
If you use {{spark-submit}} in {{cluster}} mode to submit a job to a Spark
Standalone cluster, and the {{JAVA_HOME}} environment variable is set on the
submitting machine, then DriverRunner will attempt to use the _submitter's_
JAVA_HOME to launch the driver process (instead of the worker's JAVA_HOME),
which can cause the job to fail unless the submitter and worker don't have Java
installed in the same location.
This has a pretty simple fix: read JAVA_HOME from {{sys.env}} instead of
{{command.environment}}; PR pending shortly.
> DriverRunner should not read SPARK_HOME from submitter's environment
> --------------------------------------------------------------------
>
> Key: SPARK-3734
> URL: https://issues.apache.org/jira/browse/SPARK-3734
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> If you use {{spark-submit}} in {{cluster}} mode to submit a job to a Spark
> Standalone cluster, and the {{JAVA_HOME}} environment variable is set on the
> submitting machine, then DriverRunner will attempt to use the _submitter's_
> JAVA_HOME to launch the driver process (instead of the worker's JAVA_HOME),
> which can cause the job to fail unless the submitter and worker have Java
> installed in the same location.
> This has a pretty simple fix: read JAVA_HOME from {{sys.env}} instead of
> {{command.environment}}; PR pending shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
