[
https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514403#comment-14514403
]
Patrick Wendell edited comment on SPARK-7108 at 4/27/15 4:33 PM:
-----------------------------------------------------------------
Hey Marcelo,
Your interpretation of the docs was not correct. Like all other SparkConf
variables, "spark.local.dir" is relevant for both the driver and the executors.
The only caveat is that if the environment variables are set on the executors
then it will be overridden.
The change in 1.3 breaks the semantics of this in a way that will cause Spark
to silently write to the wrong directory for those who depended on the old
behavior.
was (Author: pwendell):
Hey Marcelo,
Your interpretation of the docs was not correct. Like all other SparkConf
variables, "spark.local.dir" is relevant for both the driver and the executors.
The only caveat is that if the environment variables are set on the executors
then it will be overridden.
The change in 1.3 breaks the semantics of this in a way that will cause Spark
to silently write to the wrong directory for those who depended on the old
behavior.
- Patrick
> spark.local.dir is no longer honored in Standalone mode
> -------------------------------------------------------
>
> Key: SPARK-7108
> URL: https://issues.apache.org/jira/browse/SPARK-7108
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.1, 1.3.0
> Reporter: Josh Rosen
> Priority: Critical
>
> Prior to SPARK-4834, configuring spark.local.dir in the driver would affect
> the local directories created on the executor. After this patch, executors
> will always ignore this setting in favor of directories read from
> {{SPARK_LOCAL_DIRS}}, which is set by the standalone worker based on the
> worker's own configuration and not the application configuration.
> This change impacts users who configured {{spark.local.dir}} only in their
> driver and not via their cluster's {{spark-defaults.conf}} or
> {{spark-env.sh}} files. This is an atypical use-case, since the available
> local directories / disks are a property of the cluster and not the
> application, which probably explains why this issue has not been reported
> previously.
> The correct fix might be comment + documentation improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
