Thanks! This is exactly what I was looking for.

On 9 June 2015 at 23:18, zhou weitao <[email protected]> wrote:

> BTW, we also configured spark "spark.shuffle.consolidateFiles=true" to
> optimize it.
>
> 2015-06-10 8:16 GMT+08:00 Jeff Schroeder <[email protected]>:
>
>> http://mesos.apache.org/documentation/latest/configuration/
>> <http://mesos.apache.org/documentation/latest/configuration/#slave-options>
>>
>>
>> <http://mesos.apache.org/documentation/latest/configuration/#slave-options>Look
>> under Slave Options at --gc_delay and --gc_disk_headroom
>>
>>
>>
>>
>> On Tuesday, June 9, 2015, Gary Ogden <[email protected]> wrote:
>>
>>> We're using spark 1.2 on mesos 0.21.1. We have a simple data collection
>>> job that runs every hour. But for some reason the executor files remain in
>>> the work_dir (/tmp/mesos in our case) and eventually fill up the disk.  An
>>> example path of all these executors is:
>>>
>>> /tmp/mesos/slaves/20150408-134250-239470090-5050-15979-S1/frameworks
>>>
>>> After a while we end up with a folder for every hourly run and it fills
>>> up the disk.
>>>
>>> Shouldn't these be deleted when the job completes?  In the mesos web UI,
>>> on the main page, I noticed that the completed tasks show FINISHED, but
>>> when I go to the frameworks page, pick one of the completed frameworks, the
>>> completed tasks there show a state of KILLED.  That seems weird.
>>>
>>> Is the app coded wrong? Is there a setting in mesos I need to add to
>>> clean these up?
>>>
>>>
>>>
>>
>> --
>> Text by Jeff, typos by iPhone
>>
>
>

Reply via email to