Sorry about that miscommunication, I meant to say "from the kernel OOM killer", and you're right that the OOM shouldn't be the source of the problem here (just trying to make a mental note that our accounting is wrong), OOMs are indeed expected as part of normal operation on the slave.
It's tough to say what the issue is as I don't have any experience running on newer kernels like 3.2, I would recommend augmenting the logging in the cgroups::Freezer to try to nail down at which point the slave is blocking. In particular, in Freezer::freeze() and Freezer::watchFrozen(). Also, do you have /proc/<slave_pid>/status output for the slave when it's frozen? It will be really useful to know if that is consistently the last log line you see before things lock up as well, as that helps narrow this down. On Fri, May 10, 2013 at 10:44 AM, Brenden Matthews < [email protected]> wrote: > This isn't a kernel OOM. I started the debugger and let it go overnight > with no luck. It just doesn't want to budge. I'm aware of the memory > limit issue, but I don't think that's the real problem here. Regardless of > whether we run out of memory, the whole system shouldn't freeze when trying > to kill the process. I'm not sure if this is a kernel problem or a Mesos > problem, but I suspect that the issue isn't with the kernel. Walking > through the Mesos code, I can't see anything that is obviously wrong > either. I'm using my own branch from here: > https://github.com/airbnb/mesos/tree/testing. > > My next step will be to run a bunch of slaves in GDB and then launch some > tasks. Hopefully I can stop them and get a backtrace. I thought it might > be worth asking about it here to see if anyone else has dealt with this and > knows what's going on. > > These are EC2 instances running the latest Ubuntu Precise AMIs. Here's the > `uname -a' from one: > > Linux i-b5dbc1d5 3.2.0-41-virtual #66-Ubuntu SMP Thu Apr 25 03:47:17 UTC > 2013 x86_64 x86_64 x86_64 GNU/Linux > > > > > On Fri, May 10, 2013 at 10:21 AM, Benjamin Mahler < > [email protected] > > wrote: > > > Hey Brenden, what kernel version are you running? What version of the > code > > are you running? Are you seeing this on different machines? Can you > confirm > > you're unable to attach to the slave with gdb (that would be the easiest > > way to figure out what's blocking in the slave)? Is that always the last > > line in the log before the processes get stuck? > > > > As an aside, it looks like we aren't properly setting the JVM heap size > for > > TaskTrackers in the MesosScheduler for Hadoop. We try to estimate the JVM > > memory overhead, but it was perhaps not enough to avoid kernel OOMs. > > > > > > On Fri, May 10, 2013 at 9:57 AM, Brenden Matthews < > > [email protected]> wrote: > > > > > Hey folks, > > > > > > I'm bumping in to a problem frequently. When an executor exceeds its > > > memory limit and the executor is deemed in need of execution, the > entire > > > system becomes stuck (though I'm unsure as to whether it's stuck before > > or > > > after attempting to freeze). Here's what the mesos slave log looks > like: > > > > > > I0510 04:22:37.935608 13622 cgroups_isolator.cpp:1023] OOM notifier is > > > > triggered for executor executor_Task_Tracker_446 of framework > > > > 201305100055-1471680778-5050-18199-0000 with uuid > > > > 14a21d5a-a670-40fd-91b1-54c42b9a30e0 > > > > I0510 04:22:37.935925 13622 cgroups_isolator.cpp:1068] OOM detected > for > > > > executor executor_Task_Tracker_446 of framework > > > > 201305100055-1471680778-5050-18199-0000 with uuid > > > > 14a21d5a-a670-40fd-91b1-54c42b9a30e0 > > > > I0510 04:22:37.936982 13622 cgroups_isolator.cpp:1109] Memory limit > > > > exceeded: Requested: 2570MB Used: 2570MB > > > > MEMORY STATISTICS: > > > > cache 1682083840 > > > > rss 1012756480 > > > > mapped_file 1085440 > > > > pgpgin 22863635 > > > > pgpgout 22205715 > > > > swap 0 > > > > pgfault 11881912 > > > > pgmajfault 304 > > > > inactive_anon 0 > > > > active_anon 1012756480 > > > > inactive_file 1065136128 > > > > active_file 616947712 > > > > unevictable 0 > > > > hierarchical_memory_limit 2694840320 > > > > hierarchical_memsw_limit 9223372036854775807 > > > > total_cache 1682083840 > > > > total_rss 1012756480 > > > > total_mapped_file 1085440 > > > > total_pgpgin 22863635 > > > > total_pgpgout 22205715 > > > > total_swap 0 > > > > total_pgfault 11881912 > > > > total_pgmajfault 304 > > > > total_inactive_anon 0 > > > > total_active_anon 1012756480 > > > > total_inactive_file 1065136128 > > > > total_active_file 616947712 > > > > total_unevictable 0 > > > > I0510 04:22:37.965666 13622 cgroups_isolator.cpp:620] Killing > executor > > > > executor_Task_Tracker_446 of framework > > > > 201305100055-1471680778-5050-18199-0000 > > > > I0510 04:22:37.967418 13621 cgroups.cpp:1175] Trying to freeze cgroup > > > > > > > > > > /cgroup/mesos/framework_201305100055-1471680778-5050-18199-0000_executor_executor_Task_Tracker_446_tag_14a21d5a-a670-40fd-91b1-54c42b9a30e0 > > > > > > > > > After the last line in the log, both the executing task and the mesos > > > process are stuck indefinitely. Usually the way to resolve this is > > > `killall -9 java lt-mesos-slave' or a reboot. > > > > > > If I cat /cgroup/mesos/framework.../freezer.state, it shows that it's > > > freezing. Since the entire system is stuck, I cannot attach a debugger > > to > > > either the JVM or mesos. > > > > > > Any ideas? This problem is very frustrating and I'm stuck at this > point. > > > > > >
