In the event of HWM most middlewares provide an async callback to notify about the event... does ZMQ have anything like that? I see the same "issue"/behaviour as the parent and have a very large value set for my HWM on pub and sub but i see no memory growth that i might associate with a backlog.
On 26 September 2013 17:01, Artem Vysochyn <[email protected]> wrote: > hi Renato, > > ZMQ api manual is very clear about PUB/SUB: >>> > A socket of type ZMQ_PUB is used by a publisher to distribute data. > Messages sent are distributed in a fan out fashion to all connected > peers. > When a ZMQ_PUB socket enters the mute state due to having reached the > high water mark for a subscriber, then any messages that would be sent > to the subscriber in question shall instead be dropped until the mute > state ends. > << > I'm afraid it's not correct to say "we lost messages", because it > doesn't reflect the reality. And reality in your system may be as > following: > 1.1 SUB socket(s) _are not_ connected to PUB socket. > 1.2 SUB socket(s) _were connected_ but then disconnected from PUB socket. > 1.3 SUB socket(s) reached HWM. > > > You had said that : >> When we loose messages we have no means to > verify this event. << > Actually, you have means -- via poller. Create a poller, register > your PUB socket for OUT-going events, then poll(), then if > poller.pollout() == true then send a message. > At least this works for me. I'm on java, using jzmq JNI binding. Works > nicely. > > > 2013/9/26 gonzalo diethelm <[email protected]> >> >> Hi Renato, >> >> >> >> Your question falls under the broad concept of “reliability”. It seems >> pub/sub might not be a good fit for our requirements. Whether that is the >> case or not, if you want to get close to “no message loss” (or to detect >> when you have lost a message), you will need to add some mechanisms to your >> implementation (unique ids, increasing timestamps, idempotency, etc.). >> >> >> >> Vast, vast ground for discussion… Good luck! >> >> >> >> -- >> >> Gonzalo Diethelm >> >> DCV Chile >> >> >> >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Renato Samperio >> Sent: Thursday, September 26, 2013 11:42 AM >> To: [email protected] >> Subject: [zeromq-dev] PUB/SUB cached queue list >> >> >> >> Dear all, >> >> I have been trying to follow the list messages over the last months except >> for last two weeks. My apologies if I have skipped this issue in any message. >> >> We have written our solution in ZMQ and we have tested it with a PUB/SUB >> pattern implementation. >> >> The big trouble we have is that depending on the system hardware >> architecture (speed and memory resources) we can loose messages. When we >> loose messages we have no means to verify this event. >> >> By the way, for the requirements of our system we must not loose messages or >> have delays in their delivery. >> >> ZMQ does not have any support for checking the queue status (at both PUB and >> SUBs sides), or whenever we publish a message if this message is lost >> (because ZMQ silently throws away it) when the queue is full. >> >> Is there a way to? >> - know status of the queues, at any time; >> - get back the info that a message is/is-not queued whenever a publish is >> done. >> >> >> We would also like to set a ticket for such issues. Could you point us where >> is the right place to do it? >> >> Thanks for your help... >> >> >> Regards, >> >> Renato. >> >> ________________________________ >> >> Declaración de confidencialidad: Este Mensaje esta destinado para el uso de >> la o las personas o entidades a quien ha sido dirigido y puede contener >> información reservada y confidencial que no puede ser divulgada, difundida, >> ni aprovechada en forma alguna. El uso no autorizado de la información >> contenida en este correo podrá ser sancionado de conformidad con la ley >> chilena. Si usted ha recibido este correo electrónico por error, le pedimos >> eliminarlo junto con los archivos adjuntos y avisar inmediatamente al >> remitente, respondiendo este mensaje. Disclosure: This Message is to be used >> by the individual, individuals or entities that it is addressed to and may >> include private and confidential information that may not be disclosed, made >> public nor used in any way at all. Unauthorized use of the information in >> this electronic mail message may be subject to the penalties set forth by >> Chilean law. If you have received this electronic mail message in error, we >> ask you to destroy the message and its attached file(s) and to immediately >> notify the sender by answering this message. >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Gerry Steele _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
