[
https://issues.apache.org/jira/browse/SPARK-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-16110.
------------------------------------
Resolution: Fixed
Assignee: Kevin Grealish
Fix Version/s: 2.1.0
> Can't set Python via spark-submit for YARN cluster mode when PYSPARK_PYTHON &
> PYSPARK_DRIVER_PYTHON are set
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-16110
> URL: https://issues.apache.org/jira/browse/SPARK-16110
> Project: Spark
> Issue Type: Bug
> Components: PySpark, YARN
> Affects Versions: 1.6.1
> Environment: Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-38-generic x86_64),
> Spark 1.6.1, Azure HDInsight 3.4)
> Reporter: Kevin Grealish
> Assignee: Kevin Grealish
> Labels: easyfix
> Fix For: 2.1.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When a cluster has PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment
> variables set (needed for using non-system Python e.g.
> /usr/bin/anaconda/bin/python), then you are unable to override this per
> submission in YARN cluster mode.
> When using spark-submit (in this case via LIVY) to submit with an override:
> spark-submit --master yarn --deploy-mode cluster --conf
> 'spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=python3' --conf'
> 'spark.yarn.appMasterEnv.PYSPARK_PYTHON=python3' probe.py
> the environment variable values will override the conf settings. A workaround
> for some can be to unset the env vars but that is not always possible (e.g.
> submitting batch via LIVY where you can only pass through the parameters to
> spark-submit).
> Expectation is that the conf values above override the environment variables.
> Fix is to change the order of application of conf and env vars in the yarn
> client.
> Related discussion:https://issues.cloudera.org/browse/LIVY-159
> Backporting this to 1.6 would be great and unblocking for me.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
