Hi Jens, This problem looks very similar to the one fixed by this: https://github.com/zeromq/libzmq/pull/1649
What version of ZeroMQ do you use? Does it include this fix? Cheers, Fedor > On 21 Jun 2016, at 21:16, Jens Auer <[email protected]> wrote: > > HI, > > to add some more interesting facts, I decided to pin threads to input/output > by using the thread affinity option. After assigning one thread to the input > socket and a set of 3 threads for output connection, the situation improved > and the blocking calls got less, but I can still see them. Any idea what this > means? > > Cheers, > Jens > > Von: zeromq-dev [mailto:[email protected]] Im Auftrag von > Jens Auer > Gesendet: Dienstag, 21. Juni 2016 22:00 > An: 'ZeroMQ development list' > Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM > > Hi all, > > I did some more investigations on this, but I still don’t understand the > behavior I see. I have added some debugging output in ZeroMQ to print the > size of the inbox when reaching the watermark, and I can confirm that this is > actually happening. Everything works as expected when I set the HWM to 0, but > I cannot do this because then a slow client would overflow the system. > > Just let me recapitulate my setting. I send data with ~8500 messages per > second, all of size 1129 bytes. This should be ~100000bps. From time to time, > the sending process generates a burst where it sends an additional package of > 5s * 8500 messages from a buffer. My process receives the messages just fine > on a SUB socket, and forwards them to external clients on a ZMQ_STREAM > socket. I have set the HWM to a ridiculously large value of 100 * rate, so it > should be large enough to compensate the burst. Even when taking into account > that the actual HWM can be smaller (the doc says 60%) and that I need to send > the id message for every message, I should be able to buffer 25*rate > messages. However, I sometimes see that sending would block and I lose the > data because I send non-blocking. I think that 100 * rate should be basically > the same as HWM=0, but this does not seem to be the case. > > When running without HWM, I expected to see some increase in memory > consumption compared to the other settings, but that was not the case. This > is strange, because now it can store as many frames as needed, and since the > other settings block repeatedly, this should be more and probably grow over > time. The memory consumption is the same regardless of the buffer settings. > > Cheers, > Jens > > > Von: zeromq-dev [mailto:[email protected]] Im Auftrag von > Doron Somech > Gesendet: Samstag, 18. Juni 2016 19:04 > An: ZeroMQ development list > Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM > > It is correct, however when the socket is in shutdown process you can also > get an EAGAIN. > > Which of the high watermark do you set, send or receive? Try to set both, it > should be the send. > > On Sat, Jun 18, 2016 at 7:54 PM, Jens Auer <[email protected]> wrote: > Hi, > > I’m using the C++ binding from > https://github.com/zeromq/cppzmq/blob/master/zmq.hpp. It is quite simple. The > send method I use (there are several depending on the parameters) is: > > inline bool send (message_t &msg_, int flags_ = 0) > { > int nbytes = zmq_msg_send (&(msg_.msg), ptr, flags_); > if (nbytes >= 0) > return true; > if (zmq_errno () == EAGAIN) > return false; > throw error_t (); > } > > In this case, the return value is false for EAGAIN and true otherwise. Other > errors are handled as exceptions. > > I jut to ask under which circumstances EAGAIN is return for ZMQ_STREAM > sockets. The HWM should be large enough for anything I send, but I will > investigate more on Monday. Is it correct that HWM=0 disabled the limit > completely and I should not expect any EAGAIN ever? > > Cheers, > Jens > > Von: zeromq-dev [mailto:[email protected]] Im Auftrag von > Doron Somech > Gesendet: Samstag, 18. Juni 2016 18:47 > > An: ZeroMQ development list > Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM > > Which language and binding are you using? > > you are checking the return number from the send method, in most binding that > return the number of bytes send and not the error code. It might be different > with binding you are using but I'm not sure. > > Anyway if that is the case it might be that the number of bytes just equal > EAGAIN? > > On Sat, Jun 18, 2016 at 11:59 AM, Jens Auer <[email protected]> wrote: > Hi Doron, > > I don’t think it is disconnect. I am testing it on the local network, so > there is not much internet infrastructure in between. I am using a pseudo > client which logs socket errors, connects and disconnects and I don’t see > anything. On the server side, I have a socket monitor which logs socket > events but this does not show anything. Finally, if the TCP connection > disconnects the ZMQ_STREAM socket shall create the empty disconnect message > which I do not receive, neither before or after the EAGAIN. > > Best wishes, > Jens > > Von: zeromq-dev [mailto:[email protected]] Im Auftrag von > Doron Somech > Gesendet: Freitag, 17. Juni 2016 22:14 > An: ZeroMQ development list > Betreff: Re: [zeromq-dev] EAGAIN on ZMQ_STREAM > > Why you can exclude the first case? TCP connection can get disconnect quite > often. > Anyway from what you are describing it seems the connection got disconnected. > > On Fri, Jun 17, 2016 at 8:05 PM, Jens Auer <[email protected]> wrote: > Hi, > > I am witnessing something in my system that I didn't expect. I am processing > data at a data rate of 100Mbps, with occasional peaks of 5s of data (=5 * > 100Mbps) delivered as fast as possible because an internal buffer is > flushed. I use ZeroMQ internally, but I have to deliver the data on a TCP > plain socket at the endpoint of my system. Here, I use a ZMQ_STREAM socket > where clients connect. Basically, I keep track of connected clients and send > the data to all clients as a copy of the message. The main loop here looks > like > for(auto const& c: clientIdMessages) { > bool eagain = !socket.send(c.copy(), ZMQ_DONTWAIT | ZMQ_SNDMORE); > if (!eagain) { > eagain = !socket.send(msg.copy(), ZMQ_DONTWAIT | ZMQ_SNDMORE); > } > } > > This usually works fine, but from time to time a message is skipped because > send returns with EAGAIN. If I understand the docs correctly, this happens > if the peer is unknown (ZMQ_STREAM behavior) or when the message cannot be > queued. I can exclude the first case, so I must a high HWM for sending > messages. However, I tried to increase the HWM to a huge value of 50 times > the expected message rate and it did not solve the problem. I also tried to > disable the HWM by setting it to 0, which means "no limit". Still, I am > getting an EAGAIN which I don't expect. > > I can exclude any previous loss because I have counters showing me that the > full data stream is delivered to the ZMQ_STREAM socket code above. I also > added an assertion on !eagain after sending, and this fires. Are there any > other conditions when I can get EAGAIN from a ZMQ_STREAM socket? > > Best wishes, > Jens > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > Virenfrei. www.avast.com > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
