[
https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514862#comment-14514862
]
Patrick Wendell commented on SPARK-7108:
----------------------------------------
I see - when I wrote that, I meant it to mean "If present, SPARK_LOCAL_DIRS
will override 'spark.local.dir'" I did not mean to say "You should always
expect SPARK_LOCAL_DIRS to be present and override this". I do see that it
could be reasonably interpreted in either way.
We also specify SPARK_LOCLA_DIRS as optional and not required in the
environment file:
https://github.com/apache/spark/blob/master/conf/spark-env.sh.template#L17
Anyways I'm fine to just clarify this in the docs and more officially explain
"spark.local.dir" cannot be set in this way anymore. If many other users hit
this we can revisit the issue and see whether we want to support this for
legacy reasons, but that actually seems pretty difficult given the way this
currently works.
> spark.local.dir is no longer honored in Standalone mode
> -------------------------------------------------------
>
> Key: SPARK-7108
> URL: https://issues.apache.org/jira/browse/SPARK-7108
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.1, 1.3.0
> Reporter: Josh Rosen
> Priority: Critical
>
> Prior to SPARK-4834, configuring spark.local.dir in the driver would affect
> the local directories created on the executor. After this patch, executors
> will always ignore this setting in favor of directories read from
> {{SPARK_LOCAL_DIRS}}, which is set by the standalone worker based on the
> worker's own configuration and not the application configuration.
> This change impacts users who configured {{spark.local.dir}} only in their
> driver and not via their cluster's {{spark-defaults.conf}} or
> {{spark-env.sh}} files. This is an atypical use-case, since the available
> local directories / disks are a property of the cluster and not the
> application, which probably explains why this issue has not been reported
> previously.
> The correct fix might be comment + documentation improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
