It cleans the work dir, and SPARK_LOCAL_DIRS should be cleaned automatically.
From the source code comments:
// SPARK_LOCAL_DIRS environment variable, and deleted by the Worker when the
// application finishes.
On 13.04.2015, at 11:26, Guillaume Pitel guillaume.pi...@exensa.com wrote:
Does
That’s true, spill dirs don’t get cleaned up when something goes wrong. We are
are restarting long running jobs once in a while for cleanups and have
spark.cleaner.ttl set to a lower value than the default.
On 14.04.2015, at 17:57, Guillaume Pitel guillaume.pi...@exensa.com wrote:
Right, I
@gmail.com]
Sent: Tuesday, April 14, 2015 12:27 PM
To: Guillaume Pitel
Cc: user@spark.apache.org
Subject: Re: Is the disk space in SPARK_LOCAL_DIRS cleanned up?
That's true, spill dirs don't get cleaned up when something goes wrong. We are
are restarting long running jobs once in a while for cleanups
Right, I remember now, the only problematic case is when things go bad
and the cleaner is not executed.
Also, it can be a problem when reusing the same sparkcontext for many runs.
Guillaume
It cleans the work dir, and SPARK_LOCAL_DIRS should be cleaned
automatically. From the source code
I have set SPARK_WORKER_OPTS in spark-env.sh for that. For example:
export SPARK_WORKER_OPTS=-Dspark.worker.cleanup.enabled=true
-Dspark.worker.cleanup.appDataTtl=seconds
On 11.04.2015, at 00:01, Wang, Ningjun (LNG-NPV)
ningjun.w...@lexisnexis.com wrote:
Does anybody have an answer for
Does it also cleanup spark local dirs ? I thought it was only cleaning
$SPARK_HOME/work/
Guillaume
I have set SPARK_WORKER_OPTS in spark-env.sh for that. For example:
export SPARK_WORKER_OPTS=-Dspark.worker.cleanup.enabled=true
-Dspark.worker.cleanup.appDataTtl=seconds
On 11.04.2015, at
Hi,
I had to setup a cron job for cleanup in $SPARK_HOME/work and in
$SPARK_LOCAL_DIRS.
Here are the cron lines. Unfortunately it's for *nix machines, I guess
you will have to adapt it seriously for Windows.
12 * * * * find $SPARK_HOME/work -cmin +1440 -prune -exec rm -rf {} \+
32 * * * *
Does anybody have an answer for this?
Thanks
Ningjun
From: Wang, Ningjun (LNG-NPV)
Sent: Thursday, April 02, 2015 12:14 PM
To: user@spark.apache.org
Subject: Is the disk space in SPARK_LOCAL_DIRS cleanned up?
I set SPARK_LOCAL_DIRS to C:\temp\spark-temp. When RDDs are shuffled, spark
