Reinis, --gc_delay have a try
2015-05-22 19:24 GMT+08:00 Reinis <[email protected]>: > Hi, > > I am using Mesos 0.22.1 with Spark 1.3.1-hadoop2.4. > > I am submitting 9 spark jobs every hour with spark-submit sequentially and > they are being run by mesos as frameworks. > > The challenge is that my own application containing spark jobs (jar file) > is 160MB, spark-1.3.1-bin-hadoop2.4.tgz executor is 241 MB and unpacked > spark is another 200 MB. > > Thus every hour a terminated framework Sandboxes build up worth of 9 * > (160 + 240 + 200) = 5GB! > > I am running out of the disk space every night and am trying to somehow > garbage collect those large jar and tgz files but KEEP the log files > (stderr, stdout). > > Is it possible to somehow selectively garbage collect files stored in > terminated frameworks? > > thx > reinis > -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
