Hello, I have a 4 nodes spark cluster running on EC2 and it's running out of space in disk. I'm running Spark 1.3.1.
I have mounted a second SSD disk in every instance on /tmp/spark and set SPARK_LOCAL_DIRS and SPARK_WORKER_DIRS pointing to this folder: set | grep SPARK SPARK_LOCAL_DIRS=/tmp/spark SPARK_WORKER_DIR=/tmp/spark Once I start my cluster I can see that the Master get these variables and put everything in /tmp/spark but the workers are still using /tmp/ to spill data to the disk, what ends up filling the disk. I also tried starting the workers with -d /tmp/spark and this only moves a small file (app-...) from /opt/spark/work to my temp folder. The folders and files I can still find in /tmp/ looks like: spark-39fa5e41-3ce4-40e9-b2a7-8f3739db604e I don't know if I am missing something, any help would be much appreciated. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-WORKER-DIR-and-SPARK-LOCAL-DIR-tp23754.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
