[
https://issues.apache.org/jira/browse/SPARK-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376061#comment-14376061
]
Thomas Graves commented on SPARK-6469:
--------------------------------------
Are you saying they are not set on the driver node in yarn client mode? If so
that is what I would expect since the driver is not running on the YARN
cluster, its running on the gateway (wherever you launch it). Is the driver
now chosing local directories for the executors to use? If not what problem is
this causing exactly?
> Local directories configured for YARN are not used in yarn-client mode
> ----------------------------------------------------------------------
>
> Key: SPARK-6469
> URL: https://issues.apache.org/jira/browse/SPARK-6469
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Christophe PRÉAUD
> Priority: Minor
> Attachments: TestYarnVars.scala
>
>
> According to the [Spark YARN doc
> page|http://spark.apache.org/docs/latest/running-on-yarn.html#important-notes],
> Spark executors will use the local directories configured for YARN, not
> spark.local.dir which should be ignored.
> If this works correctly in yarn-cluster mode, I've found out that it is not
> the case in yarn-client mode.
> The problem seems to originate in the method
> [isRunningInYarnContainer|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L686].
> Indeed, I've checked with a simple application that the {{CONTAINER_ID}}
> environment variable is correctly set in yarn-cluster mode (to something like
> {{container_1426666761810_0151_01_000001}}, but not in yarn-client mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
