Set `hadoop.tmp.dir` in `spark-env.sh` solved the problem. Spark job no longer writes tmp files in /tmp/hadoop-root/.
SPARK_JAVA_OPTS+=" -Dspark.local.dir=/mnt/spark,/mnt2/spark -Dhadoop.tmp.dir=/mnt/ephemeral-hdfs" export SPARK_JAVA_OPTS I'm wondering if we need to permanently add this in the spark-ec2 script. Writing lots of tmp files in the 8GB `/` is not a great idea. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/No-space-left-on-device-error-when-pulling-data-from-s3-tp5450p5518.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
