Re: [zeromq-dev] zeromq / pub sub lost messages
The publisher does not know when a subscriber congests, as the pub-sub pattern is stateless in that regard. What you could do is to check which message was the last one received at the subscriber and request all messages from that message onwards from the publisher. However, that requires a separate connection to the publisher. The pub-sub-pattern is mostly only a viable option when the subscriber is able to work even if it doesn't receive all messages. If your usecase does not allow that you might be better of using another pattern, like req-rep or broker. Best, alex. On 04.01.2017 10:43, Philippe Guibert wrote: > On Wed, Jan 4, 2017 at 10:34 AM, Jens Auer wrote: > > Hello Jens, > > Best wishes too, > >> Hi, >> >> >> >> how do you send the messages? > > I use zmq_msg_send() procedure. > I also used zmq_send() procedure with and without MSG_NO_WAIT flag. > I expect that return code sends me an error when dropping message, > which is not the case. > >> The default HWM is 1000 messages and sending > > I used tcp transport connection, and wireshark. I observed that the > packets number sent was matching the received messages on the > subscriber side. > I deduced the problem was due to an emission problem, not a reception problem. > > I did a counter test by setting ZMQ_SNDHWM to a big value ( in my case 6). > I could observe no drops with both ipc and tcp transport connection. > >> is done asynchronously in the IO thread. If you exceed this number of >> messages in the queue, e.g. by sending a batch of much more than 1000 >> messages you will experience message drop. > > How can I know there are some drop messages ? > If I make a limitation, I expect to handle the flow congestion, in > order to resubmit to publisher, but it seems publisher does not return > me valuable error code. > > Best Regards, > > Philippe > >> >> >> >> Best wishes, >> >> Jens >> >> >> >> Von: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] Im Auftrag von >> Philippe Guibert >> Gesendet: Mittwoch, 4. Januar 2017 06:51 >> An: zeromq-dev@lists.zeromq.org >> Betreff: [zeromq-dev] zeromq / pub sub lost messages >> >> >> >> Hi all, >> >> I am using zeromq 4.1.3 version through PUB/SUB operation. >> >> Some information about the way it is implemented: >> >> - It uses one single PUB application, and single SUB application, in >> separate process. >> >> - Underlying Transport layer is IPC ( I am on the same machine). >> >> - I checked that subscription is done far before the first publication >> message is sent. >> >> - Message carried are 120 byte size. >> >> - I use zmq_msg_recv() in reception and zmq_msg_send() in transmission. >> >> - I check for return code on both sides. >> >> I have one PUB application that sends several thousand of messages per >> second, and I expect that the SUB application will receive all messages. >> >> The observation is that while the publisher sends 15000 messages, the >> subscriber can receive sometimes 15000, but other times 14750, or 11000. It >> does not seem to be deterministic. >> >> I have no error codes related to the zmq_msg_recv() or zmq_msg_send() API. >> >> Do you have any suggestions to give on how to handle that issue ? >> >> Do you suggest to use pgm protocol instead of ipc ? >> >> Are there any tracked issues related to that issue described ? >> >> Thanks in advance for any suggestion, >> >> Best Regards, >> >> Philippe >> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zeromq / pub sub lost messages
On Wed, Jan 4, 2017 at 10:34 AM, Jens Auer wrote: Hello Jens, Best wishes too, > Hi, > > > > how do you send the messages? I use zmq_msg_send() procedure. I also used zmq_send() procedure with and without MSG_NO_WAIT flag. I expect that return code sends me an error when dropping message, which is not the case. >The default HWM is 1000 messages and sending I used tcp transport connection, and wireshark. I observed that the packets number sent was matching the received messages on the subscriber side. I deduced the problem was due to an emission problem, not a reception problem. I did a counter test by setting ZMQ_SNDHWM to a big value ( in my case 6). I could observe no drops with both ipc and tcp transport connection. > is done asynchronously in the IO thread. If you exceed this number of > messages in the queue, e.g. by sending a batch of much more than 1000 > messages you will experience message drop. How can I know there are some drop messages ? If I make a limitation, I expect to handle the flow congestion, in order to resubmit to publisher, but it seems publisher does not return me valuable error code. Best Regards, Philippe > > > > Best wishes, > > Jens > > > > Von: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] Im Auftrag von > Philippe Guibert > Gesendet: Mittwoch, 4. Januar 2017 06:51 > An: zeromq-dev@lists.zeromq.org > Betreff: [zeromq-dev] zeromq / pub sub lost messages > > > > Hi all, > > I am using zeromq 4.1.3 version through PUB/SUB operation. > > Some information about the way it is implemented: > > - It uses one single PUB application, and single SUB application, in > separate process. > > - Underlying Transport layer is IPC ( I am on the same machine). > > - I checked that subscription is done far before the first publication > message is sent. > > - Message carried are 120 byte size. > > - I use zmq_msg_recv() in reception and zmq_msg_send() in transmission. > > - I check for return code on both sides. > > I have one PUB application that sends several thousand of messages per > second, and I expect that the SUB application will receive all messages. > > The observation is that while the publisher sends 15000 messages, the > subscriber can receive sometimes 15000, but other times 14750, or 11000. It > does not seem to be deterministic. > > I have no error codes related to the zmq_msg_recv() or zmq_msg_send() API. > > Do you have any suggestions to give on how to handle that issue ? > > Do you suggest to use pgm protocol instead of ipc ? > > Are there any tracked issues related to that issue described ? > > Thanks in advance for any suggestion, > > Best Regards, > > Philippe > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zeromq / pub sub lost messages
Hi, how do you send the messages? The default HWM is 1000 messages and sending is done asynchronously in the IO thread. If you exceed this number of messages in the queue, e.g. by sending a batch of much more than 1000 messages you will experience message drop. Best wishes, Jens Von: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] Im Auftrag von Philippe Guibert Gesendet: Mittwoch, 4. Januar 2017 06:51 An: zeromq-dev@lists.zeromq.org Betreff: [zeromq-dev] zeromq / pub sub lost messages Hi all, I am using zeromq 4.1.3 version through PUB/SUB operation. Some information about the way it is implemented: - It uses one single PUB application, and single SUB application, in separate process. - Underlying Transport layer is IPC ( I am on the same machine). - I checked that subscription is done far before the first publication message is sent. - Message carried are 120 byte size. - I use zmq_msg_recv() in reception and zmq_msg_send() in transmission. - I check for return code on both sides. I have one PUB application that sends several thousand of messages per second, and I expect that the SUB application will receive all messages. The observation is that while the publisher sends 15000 messages, the subscriber can receive sometimes 15000, but other times 14750, or 11000. It does not seem to be deterministic. I have no error codes related to the zmq_msg_recv() or zmq_msg_send() API. Do you have any suggestions to give on how to handle that issue ? Do you suggest to use pgm protocol instead of ipc ? Are there any tracked issues related to that issue described ? Thanks in advance for any suggestion, Best Regards, Philippe ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] zeromq / pub sub lost messages
Hi all, I am using zeromq 4.1.3 version through PUB/SUB operation. Some information about the way it is implemented: - It uses one single PUB application, and single SUB application, in separate process. - Underlying Transport layer is IPC ( I am on the same machine). - I checked that subscription is done far before the first publication message is sent. - Message carried are 120 byte size. - I use zmq_msg_recv() in reception and zmq_msg_send() in transmission. - I check for return code on both sides. I have one PUB application that sends several thousand of messages per second, and I expect that the SUB application will receive all messages. The observation is that while the publisher sends 15000 messages, the subscriber can receive sometimes 15000, but other times 14750, or 11000. It does not seem to be deterministic. I have no error codes related to the zmq_msg_recv() or zmq_msg_send() API. Do you have any suggestions to give on how to handle that issue ? Do you suggest to use pgm protocol instead of ipc ? Are there any tracked issues related to that issue described ? Thanks in advance for any suggestion, Best Regards, Philippe ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
