@gmail.com]
Sent: Tuesday, April 14, 2015 12:27 PM
To: Guillaume Pitel
Cc: user@spark.apache.org
Subject: Re: Is the disk space in SPARK_LOCAL_DIRS cleanned up?
That's true, spill dirs don't get cleaned up when something goes wrong. We are
are restarting long running jobs once in a while for cl
S="-Dspark.worker.cleanup.enabled=true
>>>> -Dspark.worker.cleanup.appDataTtl="
>>>>
>>>>> On 11.04.2015, at 00:01, Wang, Ningjun (LNG-NPV)
>>>>> mailto:ningjun.w...@lexisnexis.com>> wrote:
>>>>>
>>>>&g
nybody have an answer for this?
Thanks
Ningjun
*From:*Wang, Ningjun (LNG-NPV)
*Sent:*Thursday, April 02, 2015 12:14 PM
*To:*user@spark.apache.org <mailto:user@spark.apache.org>
*Subject:*Is the disk space in SPARK_LOCAL_DIRS cleanned up?
I set SPARK_LOCAL_DIRS to C:\temp\spark-temp. When RDDs
om: Wang, Ningjun (LNG-NPV)
>>> Sent: Thursday, April 02, 2015 12:14 PM
>>> To: user@spark.apache.org <mailto:user@spark.apache.org>
>>> Subject: Is the disk space in SPARK_LOCAL_DIRS cleanned up?
>>>
>>> I set SPARK_LOCAL_DIRS to C:\t
, at 00:01, Wang, Ningjun (LNG-NPV)
mailto:ningjun.w...@lexisnexis.com>> wrote:
Does anybody have an answer for this?
Thanks
Ningjun
*From:*Wang, Ningjun (LNG-NPV)
*Sent:*Thursday, April 02, 2015 12:14 PM
*To:*user@spark.apache.org <mailto:user@spark.apache.org>
*Subject
>
> Thanks
> Ningjun
>
> From: Wang, Ningjun (LNG-NPV)
> Sent: Thursday, April 02, 2015 12:14 PM
> To: user@spark.apache.org <mailto:user@spark.apache.org>
> Subject: Is the disk space in SPARK_LOCAL_DIRS cleanned up?
>
> I set SPARK_LOCAL_DIRS to C:\t
Thanks
Ningjun
*From:*Wang, Ningjun (LNG-NPV)
*Sent:* Thursday, April 02, 2015 12:14 PM
*To:* user@spark.apache.org
*Subject:* Is the disk space in SPARK_LOCAL_DIRS cleanned up?
I set SPARK_LOCAL_DIRS to C:\temp\spark-temp. When RDDs are
shuffled, spark writes to this folder. I found that
Does anybody have an answer for this?
Thanks
Ningjun
From: Wang, Ningjun (LNG-NPV)
Sent: Thursday, April 02, 2015 12:14 PM
To: user@spark.apache.org
Subject: Is the disk space in SPARK_LOCAL_DIRS cleanned up?
I set SPARK_LOCAL_DIRS to C:\temp\spark-temp. When RDDs are shuffled, spark
I set SPARK_LOCAL_DIRS to C:\temp\spark-temp. When RDDs are shuffled, spark
writes to this folder. I found that the disk space of this folder keep on
increase quickly and at certain point I will run out of disk space.
I wonder does spark clean up the disk spac in this folder once the shuffle
