Just a heads up that I'm going to submit pull requests to libzmq, zeromq3-x, and zeromq4-x to revert the fix for LIBZMQ-497 in order to fix LIBZMQ-576. This means some other solution needs to be found for that problem though - I don't have a clear idea of how to do that, but I do know that crashing on socket close isn't acceptable behavior.
Thanks, AJ On Wed, Nov 13, 2013 at 06:55:25PM -0600, AJ Lewis wrote: > Check out https://zeromq.jira.com/browse/LIBZMQ-576 for more info. It > looks like a previous fix for trying to ensure messages in the encoder > were sent out before socket close is causing issues. Reverting that fix > (for libzmq, it's commit f27eb67e) seems to clear this up. But we still > probably want something to fix what that commit was attempting to fix (for > details on that, see https://zeromq.jira.com/browse/LIBZMQ-497). > > AJ > > On Wed, Nov 13, 2013 at 11:06:35PM +0000, Bill M wrote: > > AJ Lewis <aj.lewis <at> quantum.com> writes: > > > > > > > > I've recently seen the same thing in 3.2.3, but hadn't been able to > > > pinpoint > > > whether the problem was in zmq proper, or in the application using it. I > > > look forward to the results of this question. > > > > > > On Wed, Nov 06, 2013 at 09:47:55AM -0800, Andy Tucker wrote: > > > > Hi, I have a program that sends messages on a ZMQ_DEALER socket with > > > > with > > > > ZMQ_DONTWAIT. If it gets back EAGAIN (perhaps because the other end is > > > > responding slowly or has gone away) it calls zmq_close to close the > > > > socket > > > > and then re-establish the connection (possibly to a new endpoint) with a > > > > new socket. ZMQ_LINGER is set to 0 (this doesn't appear to happen if > > > > ZMQ_LINGER isn't set, but that can cause other issues). > > > > > > > > I'm occasionally seeing crashes in the libzmq epoll_t thread with either > > > > "pure virtual method called" or a segmentation fault. The stack looks > > > > like > > > > (this is with libzmq 3.2.4 but others are similar): > > > > > > > > #4 0x00007f8928939ca3 in std::terminate() () from > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > > > > #5 0x00007f892893a77f in __cxa_pure_virtual () from > > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > > > > #6 0x00007f8929649db1 in zmq::v1_encoder_t::message_ready > > > > (this=0x7f8918000b90) at v1_encoder.cpp:66 > > > > #7 0x00007f892964a2a4 in > > > > zmq::encoder_base_t<zmq::v1_encoder_t>::get_data > > > > (this=0x7f8918000b90, data_=0x7f8918000928, size_=0x7f8918000930, > > > > offset_=0x0) at encoder.hpp:93 > > > > #8 0x00007f892963fb42 in zmq::stream_engine_t::out_event > > > > (this=0x7f89180008e0) at stream_engine.cpp:261 > > > > #9 0x00007f8929627d1a in zmq::epoll_t::loop (this=0x8eace0) at > > > > epoll.cpp:158 > > > > #10 0x00007f8929644996 in thread_routine (arg_=0x8ead50) at > > > > thread.cpp:83 > > > > #11 0x00007f8928be6e9a in start_thread (arg=0x7f89271b9700) at > > > > pthread_create.c:308 > > > > #12 0x00007f89293453fd in clone () at > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 > > > > > > > > Looking at the core, it appears that the memory pointed to by the > > > > msg_source field in the encoder has been freed (the "pure virtual method > > > > called" is because the vtbl pointer has been munged by something that > > > > re-allocated the buffer). The msg_source field points to the > > > > session_base_t, but that was freed by the zmq_close. The session_base_t > > > > destructor calls engine->terminate(), which would normally free the > > > > engine > > > > state but doesn't do anything if the encoder still has data left to be > > > > sent. > > > > > > > > I've reproduced this with 3.2.4, 4.0.1, and master (as of a few days > > > > ago). > > > > I filed LIBZMQ-576 and attached a small test program to the issue. > > > > > > > > This looks like a libzmq bug to me, though if I'm misusing the API in > > > > some > > > > way (or if there's a reasonable workaround) please let me know. > > > > > > > > Andy > > > > > > > _______________________________________________ > > > > zeromq-dev mailing list > > > > zeromq-dev <at> lists.zeromq.org > > > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > > I'm seeing something similar too, using zmq 3.2.3 through PHP. > > The segfault is killing the apache process with the following stack trace: > > > > #0 0x00007f4ae573ab65 in raise () from /lib/libc.so.6 > > #1 0x00007f4ae573e6b0 in abort () from /lib/libc.so.6 > > #2 0x00007f4adbaaa8c5 in __gnu_cxx::__verbose_terminate_handler() () from > > /usr/lib/libstdc++.so.6 > > #3 0x00007f4adbaa8cf6 in ?? () from /usr/lib/libstdc++.so.6 > > #4 0x00007f4adbaa8d23 in std::terminate() () from /usr/lib/libstdc++.so.6 > > #5 0x00007f4adbaa95ff in __cxa_pure_virtual () from /usr/lib/libstdc++.so.6 > > #6 0x00007f4ad92267d7 in ?? () from /usr/local/lib/libzmq.so.3 > > #7 0x00007f4ad92271af in ?? () from /usr/local/lib/libzmq.so.3 > > #8 0x00007f4ad921a0f5 in ?? () from /usr/local/lib/libzmq.so.3 > > #9 0x00007f4ad9202702 in ?? () from /usr/local/lib/libzmq.so.3 > > #10 0x00007f4ad92207fb in ?? () from /usr/local/lib/libzmq.so.3 > > #11 0x00007f4ae5a949ca in start_thread () from /lib/libpthread.so.0 > > #12 0x00007f4ae57f11cd in clone () from /lib/libc.so.6 > > #13 0x0000000000000000 in ?? () > > > > Any word on this? > > > > Thanks. > > > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > -- > AJ Lewis > Software Engineer > Quantum Corporation > > Work: 651 688-4346 > email: [email protected] > > ---------------------------------------------------------------------- > The information contained in this transmission may be confidential. Any > disclosure, copying, or further distribution of confidential information is > not permitted unless such privilege is explicitly granted in writing by > Quantum. Quantum reserves the right to have electronic communications, > including email and attachments, sent across its networks filtered through > anti virus and spam software programs and retain such messages in order to > comply with applicable data security and retention requirements. Quantum is > not responsible for the proper and complete transmission of the substance of > this communication or for any delay in its receipt. > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- AJ Lewis Software Engineer Quantum Corporation Work: 651 688-4346 email: [email protected] _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
