[
https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512280#comment-14512280
]
Patrick Wendell edited comment on SPARK-7108 at 4/25/15 6:01 AM:
-----------------------------------------------------------------
Hey I think [~joshrosen] actually worded this a bit confusingly. The issue is
that even if SPARK_LOCAL_DIRS is not set at all by the user, the setting of
"spark.local.dir" is not used from the application. This regresses from earlier
versions of spark which (as the documentation implies) would respect
spark.local.dir if set.
was (Author: pwendell):
Hey I think [~joshrosen] actually miswrote this. The issue is that even if
SPARK_LOCAL_DIRS is not set at all, the setting of "spark.local.dir" is not
used from the driver.
> Setting spark.local.dir in driver no longer overrides the standalone worker's
> local directory setting
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-7108
> URL: https://issues.apache.org/jira/browse/SPARK-7108
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.1, 1.3.0
> Reporter: Josh Rosen
> Priority: Critical
>
> Prior to SPARK-4834, configuring spark.local.dir in the driver would affect
> the local directories created on the executor. After this patch, executors
> will always ignore this setting in favor of directories read from
> {{SPARK_LOCAL_DIRS}}, which is set by the standalone worker based on the
> worker's own configuration and not the application configuration.
> This change impacts users who configured {{spark.local.dir}} only in their
> driver and not via their cluster's {{spark-defaults.conf}} or
> {{spark-env.sh}} files. This is an atypical use-case, since the available
> local directories / disks are a property of the cluster and not the
> application, which probably explains why this issue has not been reported
> previously.
> The correct fix might be comment + documentation improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
