Josh Rosen created SPARK-2974:
---------------------------------
Summary: Utils.getLocalDir() may return non-existent
spark.local.dir directory
Key: SPARK-2974
URL: https://issues.apache.org/jira/browse/SPARK-2974
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Reporter: Josh Rosen
Priority: Blocker
The patch for [SPARK-2324] modified Spark to ignore a certain number of invalid
local directories. Unfortunately, the {{Utils.getLocalDir()}} method returns
the _first_ local directory from {{spark.local.dir}}, which might not exist.
This can lead to confusing FileNotFound errors when executors attempt to fetch
files.
(I commented on this at
https://github.com/apache/spark/pull/1274#issuecomment-51537965, but I'm
opening a JIRA so we don't forget to fix it).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
