For anyone hitting the same issue, this turned out to be https://zeromq.jira.com/browse/LIBZMQ-496 - upgrade to libzmq >= 3.2.3 to avoid it.
If you're hitting this then you may also hit https://zeromq.jira.com/browse/LIBZMQ-39 (we did). That requires a much more recent version. Using stock zmq packages from stable distros seems to be a bad idea! John On Fri, Apr 4, 2014 at 12:30 PM, John Spray <[email protected]> wrote: > Hi, > > I'm wondering if anyone could provide insight into a stack trace that I've > got from a periodically (once every few days) dying process. > > Running inside the salt-minion process of SaltStack ( > https://github.com/saltstack/salt/issues/11710) > ZMQ version 3.2.2 (the one built into python-zmq on ubuntu precise) > > Without any prior knowledge of the zmq code, one suspicion would be that a > socket is in use in the signaler_t::wait call while ~mailbox_t is trying to > delete the signaler. > > Trace follows. > > Thanks, > John > > Thread 5 (Thread 0x7fb3ff2f5700 (LWP 23978)): > #0 0x00007fb40d2a0a93 in epoll_wait () from > /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fb40b48f2e7 in zmq::epoll_t::loop() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #2 0x00007fb40b484a46 in ?? () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #3 0x00007fb40e4c6e9a in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007fb40d2a03fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > #5 0x0000000000000000 in ?? () > > Thread 4 (Thread 0x7fb40e8f2700 (LWP 11303)): > #0 0x00007fb40d294a43 in poll () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fb40b47c7d2 in zmq::signaler_t::wait(int) () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #2 0x00007fb40b47a3ba in zmq::mailbox_t::recv(zmq::command_t*, int) () > from /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #3 0x00007fb40b47380b in zmq::ctx_t::terminate() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #4 0x00007fb40a5174eb in ?? () from > /usr/lib/python2.7/dist-packages/zmq/core/context.so > #5 0x000000000054c67d in PyEval_EvalFrameEx () > #6 0x000000000054c272 in PyEval_EvalFrameEx () > #7 0x0000000000575d92 in PyEval_EvalCodeEx () > #8 0x0000000000577ab0 in ?? () > #9 0x00000000004d91b6 in PyObject_Call () > #10 0x00000000004c91fa in ?? () > #11 0x00000000004d91b6 in PyObject_Call () > #12 0x00000000004d97c6 in PyEval_CallObjectWithKeywords () > #13 0x0000000000501bf5 in ?? () > #14 0x0000000000501fae in ?? () > #15 0x0000000000425d02 in ?? () > #16 0x0000000000575db4 in PyEval_EvalCodeEx () > #17 0x000000000054c028 in PyEval_EvalFrameEx () > #18 0x0000000000575d92 in PyEval_EvalCodeEx () > #19 0x000000000054c028 in PyEval_EvalFrameEx () > #20 0x000000000054c272 in PyEval_EvalFrameEx () > #21 0x000000000054c272 in PyEval_EvalFrameEx () > #22 0x0000000000575d92 in PyEval_EvalCodeEx () > #23 0x00000000004c1352 in PyRun_SimpleFileExFlags () > #24 0x00000000004c754f in Py_Main () > #25 0x00007fb40d1cd76d in __libc_start_main () from > /lib/x86_64-linux-gnu/libc.so.6 > #26 0x000000000041ba41 in _start () > > Thread 3 (Thread 0x7fb4073b5700 (LWP 11353)): > #0 0x00007fb40d2a0a93 in epoll_wait () from > /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fb40b48f2e7 in zmq::epoll_t::loop() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #2 0x00007fb40b484a46 in ?? () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #3 0x00007fb40e4c6e9a in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007fb40d2a03fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > #5 0x0000000000000000 in ?? () > > Thread 2 (Thread 0x7fb407bb6700 (LWP 11352)): > #0 0x00007fb40d2a0a93 in epoll_wait () from > /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fb40b48f2e7 in zmq::epoll_t::loop() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #2 0x00007fb40b484a46 in ?? () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #3 0x00007fb40e4c6e9a in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > ---Type <return> to continue, or q <return> to quit--- > #4 0x00007fb40d2a03fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > #5 0x0000000000000000 in ?? () > > Thread 1 (Thread 0x7fb3ffaf6700 (LWP 23977)): > #0 0x00007fb40d1e2425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fb40d1e5b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007fb40b485ac9 in zmq::zmq_abort(char const*) () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #3 0x00007fb40b47a069 in zmq::mailbox_t::~mailbox_t() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #4 0x00007fb40b48a19d in zmq::socket_base_t::~socket_base_t() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #5 0x00007fb40b479ba9 in zmq::req_t::~req_t() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #6 0x00007fb40b48f37e in zmq::epoll_t::loop() () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #7 0x00007fb40b484a46 in ?? () from > /usr/lib/python2.7/dist-packages/zmq/libzmq.so > #8 0x00007fb40e4c6e9a in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #9 0x00007fb40d2a03fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > #10 0x0000000000000000 in ?? () >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
