BTW, we also configured spark "spark.shuffle.consolidateFiles=true" to
optimize it.

2015-06-10 8:16 GMT+08:00 Jeff Schroeder <[email protected]>:

> http://mesos.apache.org/documentation/latest/configuration/
> <http://mesos.apache.org/documentation/latest/configuration/#slave-options>
>
> <http://mesos.apache.org/documentation/latest/configuration/#slave-options>Look
> under Slave Options at --gc_delay and --gc_disk_headroom
>
>
>
>
> On Tuesday, June 9, 2015, Gary Ogden <[email protected]> wrote:
>
>> We're using spark 1.2 on mesos 0.21.1. We have a simple data collection
>> job that runs every hour. But for some reason the executor files remain in
>> the work_dir (/tmp/mesos in our case) and eventually fill up the disk.  An
>> example path of all these executors is:
>>
>> /tmp/mesos/slaves/20150408-134250-239470090-5050-15979-S1/frameworks
>>
>> After a while we end up with a folder for every hourly run and it fills
>> up the disk.
>>
>> Shouldn't these be deleted when the job completes?  In the mesos web UI,
>> on the main page, I noticed that the completed tasks show FINISHED, but
>> when I go to the frameworks page, pick one of the completed frameworks, the
>> completed tasks there show a state of KILLED.  That seems weird.
>>
>> Is the app coded wrong? Is there a setting in mesos I need to add to
>> clean these up?
>>
>>
>>
>
> --
> Text by Jeff, typos by iPhone
>

Reply via email to