[jira] [Commented] (SPARK-16744) spark.yarn.appMasterEnv handling assumes values should be appended
[ https://issues.apache.org/jira/browse/SPARK-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395611#comment-15395611 ] Thomas Graves commented on SPARK-16744: --- Along with this we may consider thinking about an admin type config for env variables that would be used by cluster admins. These wouldn't be overridden by user settable ones. > spark.yarn.appMasterEnv handling assumes values should be appended > -- > > Key: SPARK-16744 > URL: https://issues.apache.org/jira/browse/SPARK-16744 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 1.6.0, 1.6.1, 1.6.2, 2.0.0 > Environment: YARN >Reporter: Kevin Grealish > > When processing environment variables set via the conf setting > spark.yarn.appMasterEnv.*, if there is an existing value the new value is > appended assuming it is a CLASSPATH like value. This does not work for most > environment variables, for example PYSPARK_PYTHON. > Related: > https://issues.apache.org/jira/browse/MAPREDUCE-6491 > https://issues.apache.org/jira/browse/SPARK-16110 > https://github.com/apache/spark/pull/13824 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16744) spark.yarn.appMasterEnv handling assumes values should be appended
[ https://issues.apache.org/jira/browse/SPARK-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394812#comment-15394812 ] Kevin Grealish commented on SPARK-16744: Issue discovered when doing narrower fix for SPARK-16110 > spark.yarn.appMasterEnv handling assumes values should be appended > -- > > Key: SPARK-16744 > URL: https://issues.apache.org/jira/browse/SPARK-16744 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 1.6.0, 1.6.1, 1.6.2, 2.0.0 > Environment: YARN >Reporter: Kevin Grealish > > When processing environment variables set via the conf setting > spark.yarn.appMasterEnv.*, if there is an existing value the new value is > appended assuming it is a CLASSPATH like value. This does not work for most > environment variables, for example PYSPARK_PYTHON. > Related: > https://issues.apache.org/jira/browse/MAPREDUCE-6491 > https://issues.apache.org/jira/browse/SPARK-16110 > https://github.com/apache/spark/pull/13824 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
