BTW, we also configured spark "spark.shuffle.consolidateFiles=true" to optimize it.
2015-06-10 8:16 GMT+08:00 Jeff Schroeder <[email protected]>: > http://mesos.apache.org/documentation/latest/configuration/ > <http://mesos.apache.org/documentation/latest/configuration/#slave-options> > > <http://mesos.apache.org/documentation/latest/configuration/#slave-options>Look > under Slave Options at --gc_delay and --gc_disk_headroom > > > > > On Tuesday, June 9, 2015, Gary Ogden <[email protected]> wrote: > >> We're using spark 1.2 on mesos 0.21.1. We have a simple data collection >> job that runs every hour. But for some reason the executor files remain in >> the work_dir (/tmp/mesos in our case) and eventually fill up the disk. An >> example path of all these executors is: >> >> /tmp/mesos/slaves/20150408-134250-239470090-5050-15979-S1/frameworks >> >> After a while we end up with a folder for every hourly run and it fills >> up the disk. >> >> Shouldn't these be deleted when the job completes? In the mesos web UI, >> on the main page, I noticed that the completed tasks show FINISHED, but >> when I go to the frameworks page, pick one of the completed frameworks, the >> completed tasks there show a state of KILLED. That seems weird. >> >> Is the app coded wrong? Is there a setting in mesos I need to add to >> clean these up? >> >> >> > > -- > Text by Jeff, typos by iPhone >

