[ https://issues.apache.org/jira/browse/MESOS-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834088#comment-16834088 ]
Benjamin Mahler edited comment on MESOS-9767 at 5/6/19 6:50 PM: ---------------------------------------------------------------- Stack trace of the Mesos master when the hang was detected. Captured using gdb. {noformat} Thread 35 (Thread 0x7fa1e7e5b700 (LWP 85875)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0ae768) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0ae768) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0ae768) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0ae750) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e7e5b700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (Thread 0x7fa1e765a700 (LWP 85876)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11ff38) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11ff38) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf11ff38) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11ff20) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e765a700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (Thread 0x7fa1e6e59700 (LWP 85877)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11d988) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11d988) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf11d988) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11d970) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e6e59700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (Thread 0x7fa1e6658700 (LWP 85878)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf128758) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf128758) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf128758) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf128740) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e6658700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 0x7fa1e5e57700 (LWP 85879)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11ffc8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11ffc8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf11ffc8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11ffb0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e5e57700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (Thread 0x7fa1e5656700 (LWP 85880)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf118f18) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf118f18) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf118f18) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf118f00) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e5656700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 0x7fa1e4e55700 (LWP 85881)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf128ea8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf128ea8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf128ea8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf128e90) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e4e55700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 0x7fa1e4654700 (LWP 85882)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970 at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0145c8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0145c8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0145c8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0145b0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e4654700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 0x7fa1e3e53700 (LWP 85883)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0026f8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0026f8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0026f8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0026e0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e3e53700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (Thread 0x7fa1e3652700 (LWP 85884)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf119668) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf119668) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf119668) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf119650) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e3652700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (Thread 0x7fa1e2e51700 (LWP 85885)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf1190c8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf1190c8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf1190c8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf1190b0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e2e51700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (Thread 0x7fa1e2650700 (LWP 85886)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11d398) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11d398) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf11d398) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11d380) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e2650700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 0x7fa1e1e4f700 (LWP 85887)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf121f58) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf121f58) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf121f58) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf121f40) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e1e4f700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 0x7fa1e164e700 (LWP 85888)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf12d038) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf12d038) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf12d038) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf12d020) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e164e700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (Thread 0x7fa1e0e4d700 (LWP 85889)): #0 0x00007fa1f05f01c2 in hash_combine_impl (k=52, h=<synthetic pointer>) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:264 #1 hash_combine<char> (v=<optimized out>, seed=<synthetic pointer>) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:337 #2 hash_range<__gnu_cxx::__normal_iterator<char const*, std::basic_string<char> > > (last=..., first=52 '4') at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:351 #3 hash_value<char, std::allocator<char> > (v=...) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:410 #4 operator() (this=<optimized out>, v=...) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:486 #5 boost::hash_combine<std::string> (seed=seed@entry=@0x7fa1e0e4c770: 0, v=...) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:337 #6 0x00007fa1f06ad178 in operator() (this=0x7fa1cc02d068, taskId=...) at /mesos/include/mesos/type_utils.hpp:634 #7 _M_hash_code (this=0x7fa1cc02d068, __k=...) at /usr/include/c++/4.9/bits/hashtable_policy.h:1261 #8 std::_Hashtable<mesos::TaskID, std::pair<mesos::TaskID const, std::_List_iterator<std::pair<mesos---Type <return> to continue, or q <return> to quit--- ::TaskID, process::Owned<mesos::Task> > > >, std::allocator<std::pair<mesos::TaskID const, std::_List_iterator<std::pair<mesos::TaskID, process::Owned<mesos::Task> > > > >, std::__detail::_Select1st, std::equal_to<mesos::TaskID>, std::hash<mesos::TaskID>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::count (this=this@entry=0x7fa1cc02d068, __k=...) at /usr/include/c++/4.9/bits/hashtable.h:1336 #9 0x00007fa1f0663eb2 in count (__x=..., this=0x7fa1cc02d068) at /usr/include/c++/4.9/bits/unordered_map.h:592 #10 contains (key=..., this=0x7fa1cc02d068) at /mesos/3rdparty/stout/include/stout/hashmap.hpp:88 #11 erase (key=..., this=0x7fa1cc02d050) at /mesos/3rdparty/stout/include/stout/boundedhashmap.hpp:92 #12 mesos::internal::master::Master::__reregisterSlave(process::UPID const&, mesos::internal::ReregisterSlaveMessage&&, process::Future<bool> const&) (this=0x561dcf047380, pid=..., reregisterSlaveMessage=<unknown type in /usr/local/lib/libmesos-1.6.0.so, CU 0x30075d6, DIE 0x38a83be>, future=...) at /mesos/src/master/master.cpp:7369 #13 0x00007fa1f14d54e1 in operator() (args#0=0x561dcf048620, this=<optimized out>) at /mesos/3rdparty/libprocess/../stout/include/stout/lambda.hpp:443 #14 process::ProcessBase::consume(process::DispatchEvent&&) (this=<optimized out>, event=<optimized out>) at /mesos/3rdparty/libprocess/src/process.cpp:3577 #15 0x00007fa1f14e89b2 in serve ( event=<unknown type in /usr/local/lib/libmesos-1.6.0.so, CU 0x14b8d81b, DIE 0x14e9f25d>, this=0x561dcf048620) at /mesos/3rdparty/libprocess/include/process/process.hpp:87 #16 process::ProcessManager::resume (this=<optimized out>, process=0x561dcf048620) at /mesos/3rdparty/libprocess/src/process.cpp:3002 #17 0x00007fa1f14ee226 in operator() (__closure=0x561dcf119158) at /mesos/3rdparty/libprocess/src/process.cpp:2511 #18 _M_invoke<> (this=0x561dcf119158) at /usr/include/c++/4.9/functional:1700 #19 operator() (this=0x561dcf119158) at /usr/include/c++/4.9/functional:1688 #20 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf119140) at /usr/include/c++/4.9/thread:115 #21 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #22 0x00007fa1ee520064 in start_thread (arg=0x7fa1e0e4d700) at pthread_create.c:309 #23 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (Thread 0x7fa1e064c700 (LWP 85890)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf123148) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf123148) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf123148) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf123130) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e064c700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 0x7fa1dfe4b700 (LWP 85891)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe7e28) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe7e28) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe7e28) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe7e10) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dfe4b700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (Thread 0x7fa1df64a700 (LWP 85892)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe76e8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe76e8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe76e8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe76d0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1df64a700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (Thread 0x7fa1dee49700 (LWP 85893)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe81c8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe81c8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe81c8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe81b0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dee49700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (Thread 0x7fa1de648700 (LWP 85894)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe7a88) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe7a88) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe7a88) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe7a70) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1de648700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (Thread 0x7fa1dde47700 (LWP 85895)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf061c58) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf061c58) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf061c58) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf061c40) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dde47700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (Thread 0x7fa1dd646700 (LWP 85896)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf012fb8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf012fb8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf012fb8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf012fa0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dd646700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (Thread 0x7fa1dce45700 (LWP 85897)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf058ce8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf058ce8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf058ce8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf058cd0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dce45700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (Thread 0x7fa1dc644700 (LWP 85898)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0569d8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0569d8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0569d8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0569c0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dc644700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (Thread 0x7fa1dbe43700 (LWP 85899)): #0 0x00007fa1ee255c03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1ecba7b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 #2 0x00007fa1ecb92d5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 #3 0x00007fa1f155fbed in process::EventLoop::run () at /mesos/3rdparty/libprocess/src/libevent.cpp:98 #4 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007fa1ee520064 in start_thread (arg=0x7fa1dbe43700) at pthread_create.c:309 #6 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7fa1db2da700 (LWP 85900)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f15c4a03 in BGThread (this=0x561dcf0608e0) at util/env_posix.cc:573 #2 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x561dcf0608e0) at util/env_posix.cc:520 #3 0x00007fa1ee520064 in start_thread (arg=0x7fa1db2da700) at pthread_create.c:309 #4 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7fa1d8ad5700 (LWP 97884)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa15c033390) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d8ad5700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7fa17affd700 (LWP 97885)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa17c004050) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa17affd700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7fa17b7fe700 (LWP 97886)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa15c033390) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa17b7fe700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7fa1d9ad7700 (LWP 97887)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa17c004050) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d9ad7700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7fa1d92d6700 (LWP 97888)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa1b000e0d0) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d92d6700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7fa17bfff700 (LWP 97889)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa1b000e0d0) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa17bfff700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7fa1daad9700 (LWP 97890)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa1c800c370) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1daad9700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7fa1da2d8700 (LWP 97891)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa1c800c370) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1da2d8700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7fa1f2718800 (LWP 85829)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1ee7e7c9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #2 0x00007fa1f14e8f80 in synchronized_wait<std::condition_variable, std::mutex> ( mutex=0x561dcf0489d8, cv=0x561dcf048a00) at /mesos/3rdparty/libprocess/../stout/include/stout/synchronized.hpp:201 #3 wait (this=0x561dcf0489d0) at /mesos/3rdparty/libprocess/src/gate.hpp:50 #4 process::ProcessManager::wait (this=<optimized out>, pid=...) at /mesos/3rdparty/libprocess/src/process.cpp:3227 #5 0x00007fa1f14ebf17 in process::wait (pid=..., duration=...) at /mesos/3rdparty/libprocess/src/process.cpp:3968 #6 0x0000561dcddc05c9 in main (argc=<optimized out>, argv=<optimized out>) at /mesos/src/master/main.cpp:546 {noformat} was (Author: ggarg): Stack trace of the Mesos master when the hang was detected. Captured using gdb. Thread 35 (Thread 0x7fa1e7e5b700 (LWP 85875)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0ae768) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0ae768) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0ae768) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0ae750) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e7e5b700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (Thread 0x7fa1e765a700 (LWP 85876)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11ff38) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11ff38) at /usr/include/c++/4.9/functional:1700 ---Type <return> to continue, or q <return> to quit--- #7 operator() (this=0x561dcf11ff38) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11ff20) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e765a700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (Thread 0x7fa1e6e59700 (LWP 85877)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11d988) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11d988) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf11d988) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11d970) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e6e59700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (Thread 0x7fa1e6658700 (LWP 85878)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 ---Type <return> to continue, or q <return> to quit--- #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf128758) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf128758) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf128758) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf128740) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e6658700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 0x7fa1e5e57700 (LWP 85879)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11ffc8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11ffc8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf11ffc8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11ffb0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e5e57700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 30 (Thread 0x7fa1e5656700 (LWP 85880)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf118f18) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf118f18) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf118f18) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf118f00) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e5656700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 0x7fa1e4e55700 (LWP 85881)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf128ea8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf128ea8) at /usr/include/c++/4.9/functional:1700 ---Type <return> to continue, or q <return> to quit--- #7 operator() (this=0x561dcf128ea8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf128e90) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e4e55700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 0x7fa1e4654700 (LWP 85882)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0145c8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0145c8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0145c8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0145b0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e4654700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 0x7fa1e3e53700 (LWP 85883)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 ---Type <return> to continue, or q <return> to quit--- #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0026f8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0026f8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0026f8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0026e0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e3e53700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (Thread 0x7fa1e3652700 (LWP 85884)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf119668) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf119668) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf119668) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf119650) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e3652700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 25 (Thread 0x7fa1e2e51700 (LWP 85885)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf1190c8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf1190c8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf1190c8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf1190b0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e2e51700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (Thread 0x7fa1e2650700 (LWP 85886)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf11d398) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf11d398) at /usr/include/c++/4.9/functional:1700 ---Type <return> to continue, or q <return> to quit--- #7 operator() (this=0x561dcf11d398) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf11d380) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e2650700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 0x7fa1e1e4f700 (LWP 85887)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf121f58) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf121f58) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf121f58) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf121f40) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e1e4f700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 0x7fa1e164e700 (LWP 85888)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 ---Type <return> to continue, or q <return> to quit--- #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf12d038) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf12d038) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf12d038) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf12d020) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e164e700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (Thread 0x7fa1e0e4d700 (LWP 85889)): #0 0x00007fa1f05f01c2 in hash_combine_impl (k=52, h=<synthetic pointer>) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:264 #1 hash_combine<char> (v=<optimized out>, seed=<synthetic pointer>) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:337 #2 hash_range<__gnu_cxx::__normal_iterator<char const*, std::basic_string<char> > > (last=..., first=52 '4') at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:351 #3 hash_value<char, std::allocator<char> > (v=...) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:410 #4 operator() (this=<optimized out>, v=...) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:486 #5 boost::hash_combine<std::string> (seed=seed@entry=@0x7fa1e0e4c770: 0, v=...) at ../3rdparty/boost-1.65.0/boost/functional/hash/hash.hpp:337 #6 0x00007fa1f06ad178 in operator() (this=0x7fa1cc02d068, taskId=...) at /mesos/include/mesos/type_utils.hpp:634 #7 _M_hash_code (this=0x7fa1cc02d068, __k=...) at /usr/include/c++/4.9/bits/hashtable_policy.h:1261 #8 std::_Hashtable<mesos::TaskID, std::pair<mesos::TaskID const, std::_List_iterator<std::pair<mesos---Type <return> to continue, or q <return> to quit--- ::TaskID, process::Owned<mesos::Task> > > >, std::allocator<std::pair<mesos::TaskID const, std::_List_iterator<std::pair<mesos::TaskID, process::Owned<mesos::Task> > > > >, std::__detail::_Select1st, std::equal_to<mesos::TaskID>, std::hash<mesos::TaskID>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::count (this=this@entry=0x7fa1cc02d068, __k=...) at /usr/include/c++/4.9/bits/hashtable.h:1336 #9 0x00007fa1f0663eb2 in count (__x=..., this=0x7fa1cc02d068) at /usr/include/c++/4.9/bits/unordered_map.h:592 #10 contains (key=..., this=0x7fa1cc02d068) at /mesos/3rdparty/stout/include/stout/hashmap.hpp:88 #11 erase (key=..., this=0x7fa1cc02d050) at /mesos/3rdparty/stout/include/stout/boundedhashmap.hpp:92 #12 mesos::internal::master::Master::__reregisterSlave(process::UPID const&, mesos::internal::ReregisterSlaveMessage&&, process::Future<bool> const&) (this=0x561dcf047380, pid=..., reregisterSlaveMessage=<unknown type in /usr/local/lib/libmesos-1.6.0.so, CU 0x30075d6, DIE 0x38a83be>, future=...) at /mesos/src/master/master.cpp:7369 #13 0x00007fa1f14d54e1 in operator() (args#0=0x561dcf048620, this=<optimized out>) at /mesos/3rdparty/libprocess/../stout/include/stout/lambda.hpp:443 #14 process::ProcessBase::consume(process::DispatchEvent&&) (this=<optimized out>, event=<optimized out>) at /mesos/3rdparty/libprocess/src/process.cpp:3577 #15 0x00007fa1f14e89b2 in serve ( event=<unknown type in /usr/local/lib/libmesos-1.6.0.so, CU 0x14b8d81b, DIE 0x14e9f25d>, this=0x561dcf048620) at /mesos/3rdparty/libprocess/include/process/process.hpp:87 #16 process::ProcessManager::resume (this=<optimized out>, process=0x561dcf048620) at /mesos/3rdparty/libprocess/src/process.cpp:3002 #17 0x00007fa1f14ee226 in operator() (__closure=0x561dcf119158) at /mesos/3rdparty/libprocess/src/process.cpp:2511 #18 _M_invoke<> (this=0x561dcf119158) at /usr/include/c++/4.9/functional:1700 #19 operator() (this=0x561dcf119158) at /usr/include/c++/4.9/functional:1688 #20 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf119140) at /usr/include/c++/4.9/thread:115 ---Type <return> to continue, or q <return> to quit--- #21 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #22 0x00007fa1ee520064 in start_thread (arg=0x7fa1e0e4d700) at pthread_create.c:309 #23 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (Thread 0x7fa1e064c700 (LWP 85890)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf123148) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf123148) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf123148) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf123130) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1e064c700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 0x7fa1dfe4b700 (LWP 85891)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 ---Type <return> to continue, or q <return> to quit--- #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe7e28) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe7e28) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe7e28) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe7e10) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dfe4b700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (Thread 0x7fa1df64a700 (LWP 85892)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe76e8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe76e8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe76e8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe76d0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1df64a700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (Thread 0x7fa1dee49700 (LWP 85893)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 ---Type <return> to continue, or q <return> to quit--- #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe81c8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe81c8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe81c8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe81b0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dee49700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (Thread 0x7fa1de648700 (LWP 85894)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcefe7a88) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcefe7a88) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcefe7a88) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcefe7a70) at /usr/include/c++/4.9/thread:115 ---Type <return> to continue, or q <return> to quit--- #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1de648700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (Thread 0x7fa1dde47700 (LWP 85895)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf061c58) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf061c58) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf061c58) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf061c40) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dde47700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (Thread 0x7fa1dd646700 (LWP 85896)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 ---Type <return> to continue, or q <return> to quit--- #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf012fb8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf012fb8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf012fb8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf012fa0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dd646700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (Thread 0x7fa1dce45700 (LWP 85897)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf058ce8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf058ce8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf058ce8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf058cd0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dce45700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (Thread 0x7fa1dc644700 (LWP 85898)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 ---Type <return> to continue, or q <return> to quit--- #1 0x00007fa1f14d6e82 in wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:115 #2 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/semaphore.hpp:154 #3 wait (this=<optimized out>) at /mesos/3rdparty/libprocess/src/run_queue.hpp:73 #4 process::ProcessManager::dequeue (this=0x561dcf063970) at /mesos/3rdparty/libprocess/src/process.cpp:3305 #5 0x00007fa1f14ee22f in operator() (__closure=0x561dcf0569d8) at /mesos/3rdparty/libprocess/src/process.cpp:2505 #6 _M_invoke<> (this=0x561dcf0569d8) at /usr/include/c++/4.9/functional:1700 #7 operator() (this=0x561dcf0569d8) at /usr/include/c++/4.9/functional:1688 #8 std::thread::_Impl<std::_Bind_simple<process::ProcessManager::init_threads()::<lambda()>()> >::_M_run(void) (this=0x561dcf0569c0) at /usr/include/c++/4.9/thread:115 #9 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007fa1ee520064 in start_thread (arg=0x7fa1dc644700) at pthread_create.c:309 #11 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (Thread 0x7fa1dbe43700 (LWP 85899)): #0 0x00007fa1ee255c03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1ecba7b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 #2 0x00007fa1ecb92d5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 #3 0x00007fa1f155fbed in process::EventLoop::run () at /mesos/3rdparty/libprocess/src/libevent.cpp:98 #4 0x00007fa1ee7eb990 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007fa1ee520064 in start_thread (arg=0x7fa1dbe43700) at pthread_create.c:309 #6 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7fa1db2da700 (LWP 85900)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f15c4a03 in BGThread (this=0x561dcf0608e0) at util/env_posix.cc:573 ---Type <return> to continue, or q <return> to quit--- #2 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x561dcf0608e0) at util/env_posix.cc:520 #3 0x00007fa1ee520064 in start_thread (arg=0x7fa1db2da700) at pthread_create.c:309 #4 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7fa1d8ad5700 (LWP 97884)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa15c033390) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d8ad5700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7fa17affd700 (LWP 97885)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa17c004050) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa17affd700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7fa17b7fe700 (LWP 97886)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa15c033390) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa17b7fe700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7fa1d9ad7700 (LWP 97887)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa17c004050) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d9ad7700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 5 (Thread 0x7fa1d92d6700 (LWP 97888)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa1b000e0d0) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1d92d6700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7fa17bfff700 (LWP 97889)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa1b000e0d0) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa17bfff700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7fa1daad9700 (LWP 97890)): #0 0x00007fa1ee24caed in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fa1f17070cb in do_io (v=0x7fa1c800c370) at src/mt_adaptor.c:386 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1daad9700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7fa1da2d8700 (LWP 97891)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fa1f1707274 in do_completion (v=0x7fa1c800c370) at src/mt_adaptor.c:462 #2 0x00007fa1ee520064 in start_thread (arg=0x7fa1da2d8700) at pthread_create.c:309 #3 0x00007fa1ee25562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7fa1f2718800 (LWP 85829)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 ---Type <return> to continue, or q <return> to quit--- #1 0x00007fa1ee7e7c9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #2 0x00007fa1f14e8f80 in synchronized_wait<std::condition_variable, std::mutex> ( mutex=0x561dcf0489d8, cv=0x561dcf048a00) at /mesos/3rdparty/libprocess/../stout/include/stout/synchronized.hpp:201 #3 wait (this=0x561dcf0489d0) at /mesos/3rdparty/libprocess/src/gate.hpp:50 #4 process::ProcessManager::wait (this=<optimized out>, pid=...) at /mesos/3rdparty/libprocess/src/process.cpp:3227 #5 0x00007fa1f14ebf17 in process::wait (pid=..., duration=...) at /mesos/3rdparty/libprocess/src/process.cpp:3968 #6 0x0000561dcddc05c9 in main (argc=<optimized out>, argv=<optimized out>) at /mesos/src/master/main.cpp:546 > Add self health monitoring in Mesos master > ------------------------------------------ > > Key: MESOS-9767 > URL: https://issues.apache.org/jira/browse/MESOS-9767 > Project: Mesos > Issue Type: Task > Components: master > Affects Versions: 1.6.0 > Reporter: Gaurav Garg > Priority: Major > Fix For: 1.7.2 > > > We have seen issue where Mesos master got stuck and was not responding to > HTTP endpoints like "/metrics/snapshot". This results in calls by the > frameworks and metrics collector to the master to hang. Currently we emit > 'master alive' metric using prometheus. If master hangs, this metrics is not > published and we detect the hangs using alerts on top of this metrics. By the > time someone would have got the alert and restarted the master process, > 15-30mins would have passed by. This results in SLA violation by Mesos > cluster users. > It will be nice to implement a self health check monitoring to detect if the > Mesos master is hung/stuck. This will help us to quickly crash the master > process so that one of the other member of the quorum can acquire ZK > leadership lock. > We can use the "/master/health" endpoint for health checks. > Health checks can be initiated in > [src/master/main.cpp|[https://github.com/apache/mesos/blob/master/src/master/main.cpp]] > just after the child master process is > [spawned.|[https://github.com/apache/mesos/blob/master/src/master/main.cpp#L543]] > We can leverage the > [HealthChecker|[https://github.com/apache/mesos/blob/master/src/checks/health_checker.hpp]] > for this one. One downside is that HealthChecker currently takes TaskId as > an input which is not valid for master health check. > We can add following flags to control the self heath checking: > # self_monitoring_enabled: Whether self monitoring is enabled. > # self_monitoring_consecutive_failures: After this many number of health > failures, master is crashed. > # self_monitoring_interval_secs: Interval at which health checks are > performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)