[jira] [Created] (SPARK-2325) Utils.getLocalDir had better check the directory and choose a good one instead of choosing the first one directly

[YanTang Zhai (JIRA)]

YanTang Zhai created SPARK-2325:
-----------------------------------

             Summary: Utils.getLocalDir had better check the directory and 
choose a good one instead of choosing the first one directly
                 Key: SPARK-2325
                 URL: https://issues.apache.org/jira/browse/SPARK-2325
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
            Reporter: YanTang Zhai


If the first directory of spark.local.dir is bad, application will exit with 
the exception:
Exception in thread "main" java.io.IOException: Failed to create a temp 
directory (under /data1/sparkenv/local) after 10 attempts!
        at org.apache.spark.util.Utils$.createTempDir(Utils.scala:258)
        at 
org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:154)
        at 
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
        at 
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
        at 
org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
        at 
org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
        at JobTaskJoin$.main(JobTaskJoin.scala:9)
        at JobTaskJoin.main(JobTaskJoin.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Utils.getLocalDir had better check the directory and choose a good one instead 
of choosing the first one directly. For example, spark.local.dir is 
/data1/sparkenv/local,/data2/sparkenv/local. The disk data1 is bad while the 
disk data2 is good, we could choose the data2 not data1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)