Thanks! This is exactly what I was looking for. On 9 June 2015 at 23:18, zhou weitao <[email protected]> wrote:
> BTW, we also configured spark "spark.shuffle.consolidateFiles=true" to > optimize it. > > 2015-06-10 8:16 GMT+08:00 Jeff Schroeder <[email protected]>: > >> http://mesos.apache.org/documentation/latest/configuration/ >> <http://mesos.apache.org/documentation/latest/configuration/#slave-options> >> >> >> <http://mesos.apache.org/documentation/latest/configuration/#slave-options>Look >> under Slave Options at --gc_delay and --gc_disk_headroom >> >> >> >> >> On Tuesday, June 9, 2015, Gary Ogden <[email protected]> wrote: >> >>> We're using spark 1.2 on mesos 0.21.1. We have a simple data collection >>> job that runs every hour. But for some reason the executor files remain in >>> the work_dir (/tmp/mesos in our case) and eventually fill up the disk. An >>> example path of all these executors is: >>> >>> /tmp/mesos/slaves/20150408-134250-239470090-5050-15979-S1/frameworks >>> >>> After a while we end up with a folder for every hourly run and it fills >>> up the disk. >>> >>> Shouldn't these be deleted when the job completes? In the mesos web UI, >>> on the main page, I noticed that the completed tasks show FINISHED, but >>> when I go to the frameworks page, pick one of the completed frameworks, the >>> completed tasks there show a state of KILLED. That seems weird. >>> >>> Is the app coded wrong? Is there a setting in mesos I need to add to >>> clean these up? >>> >>> >>> >> >> -- >> Text by Jeff, typos by iPhone >> > >

