Thanks for the info. The file system in use is a Lustre file system. Best, Tassilo
On Wed, Feb 11, 2015 at 12:15 PM, Charles Feduke <charles.fed...@gmail.com> wrote: > A central location, such as NFS? > > If they are temporary for the purpose of further job processing you'll > want to keep them local to the node in the cluster, i.e., in /tmp. If they > are centralized you won't be able to take advantage of data locality and > the central file store will become a bottleneck for further processing. > > If /tmp isn't an option because you want to be able to monitor the file > outputs as they occur you can also use HDFS (assuming your Spark nodes are > also HDFS members they will benefit from data locality). > > It looks like the problem you are seeing is that a lock cannot be acquired > on the output file in the central file system. > > On Wed Feb 11 2015 at 11:55:55 AM TJ Klein <tjkl...@gmail.com> wrote: > >> Hi, >> >> Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a different >> path then local directory. >> >> On our cluster we have a folder for temporary files (in a central file >> system), which is called /scratch. >> >> When setting SPARK_LOCAL_DIRS=/scratch/<node name> >> >> I get: >> >> An error occurred while calling >> z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile. >> : org.apache.spark.SparkException: Job aborted due to stage failure: >> Task 0 >> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage >> 0.0 >> (TID 3, XXXXXXX): java.io.IOException: Function not implemented >> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) >> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:91) >> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022) >> at java.nio.channels.FileChannel.lock(FileChannel.java:1052) >> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379) >> >> Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea? >> >> Best, >> Tassilo >> >> >> >> >> >> -- >> View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >>
