I'm not 100% sure but I think it goes
like this :
spark.local.dir can and should be set both on the executors and on the driver (if the driver broadcast variables, the files will be stored in this directory) the SPARK_WORKER_DIR is where the jars and the log output of the executors is placed (default $SPARK_HOME/work/) and it should be cleaned regularly In $SPARK_HOME/logs are found the logs of the workers and master Guillaume Hi, I'm confused about the -Dspark.local.dir and SPARK_WORKER_DIR(--work-dir).What's the difference? I have set -Dspark.local.dir for all my worker nodes but I'm still seeing directories being created in /tmp when the job is running. I have also tried setting -Dspark.local.dir when I run the application. Thanks! --
|
- Spark temp dir (spark.local.dir) Tsai Li Ming
- Re: Spark temp dir (spark.local.dir) Guillaume Pitel
- Re: Spark temp dir (spark.local.dir) Guillaume Pitel
- Re: Spark temp dir (spark.local.dir) Tsai Li Ming
- Re: Spark temp dir (spark.local.dir) Guillaume Pitel
- Re: Spark temp dir (spark.local.dir) Scott Clasen