[
https://issues.apache.org/jira/browse/SPARK-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100089#comment-14100089
]
Apache Spark commented on SPARK-2974:
-------------------------------------
User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/2002
> Utils.getLocalDir() may return non-existent spark.local.dir directory
> ---------------------------------------------------------------------
>
> Key: SPARK-2974
> URL: https://issues.apache.org/jira/browse/SPARK-2974
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Blocker
>
> The patch for [SPARK-2324] modified Spark to ignore a certain number of
> invalid local directories. Unfortunately, the {{Utils.getLocalDir()}} method
> returns the _first_ local directory from {{spark.local.dir}}, which might not
> exist. This can lead to confusing FileNotFound errors when executors attempt
> to fetch files.
> (I commented on this at
> https://github.com/apache/spark/pull/1274#issuecomment-51537965, but I'm
> opening a JIRA so we don't forget to fix it).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
