Hi Tomek, Thanks for the quick reply. After digging a bit into Mesos code we were able understand that age actually mean threshold age. Anything older than the “age" would be GCed. We are going to try different setting starting with "--gc_disk_headroom=.2 --gc_delay=2hrs”. Is there any downside of the going with very low GC delay?
Thanks, Venkat > On Oct 26, 2017, at 4:28 PM, Tomek Janiszewski <jani...@gmail.com> wrote: > >> gc_delay * max(0.0, (1.0 - gc_disk_headroom - disk usage)) > > *Example:* > gc_delay = 7days > gc_disk_headroom = 0.1 > disk_usage = 0.8 > 7 * max(0.0, 1 - 0.1 - 0.8) = 7 * max(0.0, 0.1) = 0.7 days = 16 h 48 min > > Can you show some logs containging information about GC? > > pt., 27 paź 2017 o 00:43 użytkownik Venkat Morampudi < > venkatmoramp...@gmail.com <mailto:venkatmoramp...@gmail.com>> napisał: > >> Hello, >> In our production env, we noticed that our disk filled up because one >> framework had a lot of failed/completed executors folders laying around. >> The folders eventually filled up the disk. >> >> >> 228M >> /mnt/resource/slaves/c8674097-6e67-4609-b022-3e11de380fe5-S2/frameworks/35e600c2-6f43-402c-856f-9084c0040187-002/executors/52334.1.0 >> 228M >> /mnt/resource/slaves/c8674097-6e67-4609-b022-3e11de380fe5-S2/frameworks/35e600c2-6f43-402c-856f-9084c0040187-002/executors/52334.2.0 >> 228M >> /mnt/resource/slaves/c8674097-6e67-4609-b022-3e11de380fe5-S2/frameworks/35e600c2-6f43-402c-856f-9084c0040187-002/executors/52335.1.0 >> 228M >> /mnt/resource/slaves/c8674097-6e67-4609-b022-3e11de380fe5-S2/frameworks/35e600c2-6f43-402c-856f-9084c0040187-002/executors/52335.2.0 >> 228M >> /mnt/resource/slaves/c8674097-6e67-4609-b022-3e11de380fe5-S2/frameworks/35e600c2-6f43-402c-856f-9084c0040187-002/executors/52336.1.0 >> >> http://mesos.apache.org/documentation/latest/sandbox/#sandbox-lifecycle < >> http://mesos.apache.org/documentation/latest/sandbox/#sandbox-lifecycle >> <http://mesos.apache.org/documentation/latest/sandbox/#sandbox-lifecycle>> >> >> We have our lifecycle clean up set to the default which is 7days, I >> believe. >> >> We wanted to know if this is the proper way to clean up the >> failed/completed executors folders for a running framework? >> OR does the framework need to be Inactive or Completed for the garbage >> collection to work? >> OR does the framework , itself, need to deal with cleaning up its own >> executors? >> >> Bonus question: How does “gc_disk_headroom” actually work? This equation >> will always return 0 it seems. gc_delay * max(0.0, (1.0 - gc_disk_headroom >> - disk usage)) >> >> Thanks, >> Venkat
