hi Pieter, Quick question on PUB/SUB: am I right that one can detect message loss using poller? Because I extensively use poller just for that. Am I right?
2013/9/26 Pieter Hintjens <[email protected]>: > The pub-sub pattern is meant for high-volume unbreakable data > distribution. In this use case there is literally no recovery from a > slow subscriber except to drop messages. Any other strategy will > create latency spikes, or memory overflows on the publisher. > > You can lose messages at the subscriber side too, if the application > thread is too slow in pulling them off the sub socket. > > If you need guaranteed delivery you cannot use PUB-SUB. You can use > ROUTER-DEALER and add your own acknowledgements. > > There's a lot of discussion of the different strategies and tradeoffs > in the Guide. > > -Pieter > > On Thu, Sep 26, 2013 at 6:15 PM, Gerry Steele <[email protected]> wrote: >> In the event of HWM most middlewares provide an async callback to >> notify about the event... does ZMQ have anything like that? I see the >> same "issue"/behaviour as the parent and have a very large value set >> for my HWM on pub and sub but i see no memory growth that i might >> associate with a backlog. >> >> On 26 September 2013 17:01, Artem Vysochyn <[email protected]> wrote: >>> hi Renato, >>> >>> ZMQ api manual is very clear about PUB/SUB: >>>>> >>> A socket of type ZMQ_PUB is used by a publisher to distribute data. >>> Messages sent are distributed in a fan out fashion to all connected >>> peers. >>> When a ZMQ_PUB socket enters the mute state due to having reached the >>> high water mark for a subscriber, then any messages that would be sent >>> to the subscriber in question shall instead be dropped until the mute >>> state ends. >>> << >>> I'm afraid it's not correct to say "we lost messages", because it >>> doesn't reflect the reality. And reality in your system may be as >>> following: >>> 1.1 SUB socket(s) _are not_ connected to PUB socket. >>> 1.2 SUB socket(s) _were connected_ but then disconnected from PUB socket. >>> 1.3 SUB socket(s) reached HWM. >>> >>> >>> You had said that : >> When we loose messages we have no means to >>> verify this event. << >>> Actually, you have means -- via poller. Create a poller, register >>> your PUB socket for OUT-going events, then poll(), then if >>> poller.pollout() == true then send a message. >>> At least this works for me. I'm on java, using jzmq JNI binding. Works >>> nicely. >>> >>> >>> 2013/9/26 gonzalo diethelm <[email protected]> >>>> >>>> Hi Renato, >>>> >>>> >>>> >>>> Your question falls under the broad concept of “reliability”. It seems >>>> pub/sub might not be a good fit for our requirements. Whether that is the >>>> case or not, if you want to get close to “no message loss” (or to detect >>>> when you have lost a message), you will need to add some mechanisms to >>>> your implementation (unique ids, increasing timestamps, idempotency, etc.). >>>> >>>> >>>> >>>> Vast, vast ground for discussion… Good luck! >>>> >>>> >>>> >>>> -- >>>> >>>> Gonzalo Diethelm >>>> >>>> DCV Chile >>>> >>>> >>>> >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of Renato Samperio >>>> Sent: Thursday, September 26, 2013 11:42 AM >>>> To: [email protected] >>>> Subject: [zeromq-dev] PUB/SUB cached queue list >>>> >>>> >>>> >>>> Dear all, >>>> >>>> I have been trying to follow the list messages over the last months except >>>> for last two weeks. My apologies if I have skipped this issue in any >>>> message. >>>> >>>> We have written our solution in ZMQ and we have tested it with a PUB/SUB >>>> pattern implementation. >>>> >>>> The big trouble we have is that depending on the system hardware >>>> architecture (speed and memory resources) we can loose messages. When we >>>> loose messages we have no means to verify this event. >>>> >>>> By the way, for the requirements of our system we must not loose messages >>>> or have delays in their delivery. >>>> >>>> ZMQ does not have any support for checking the queue status (at both PUB >>>> and SUBs sides), or whenever we publish a message if this message is lost >>>> (because ZMQ silently throws away it) when the queue is full. >>>> >>>> Is there a way to? >>>> - know status of the queues, at any time; >>>> - get back the info that a message is/is-not queued whenever a publish is >>>> done. >>>> >>>> >>>> We would also like to set a ticket for such issues. Could you point us >>>> where is the right place to do it? >>>> >>>> Thanks for your help... >>>> >>>> >>>> Regards, >>>> >>>> Renato. >>>> >>>> ________________________________ >>>> >>>> Declaración de confidencialidad: Este Mensaje esta destinado para el uso >>>> de la o las personas o entidades a quien ha sido dirigido y puede contener >>>> información reservada y confidencial que no puede ser divulgada, >>>> difundida, ni aprovechada en forma alguna. El uso no autorizado de la >>>> información contenida en este correo podrá ser sancionado de conformidad >>>> con la ley chilena. Si usted ha recibido este correo electrónico por >>>> error, le pedimos eliminarlo junto con los archivos adjuntos y avisar >>>> inmediatamente al remitente, respondiendo este mensaje. Disclosure: This >>>> Message is to be used by the individual, individuals or entities that it >>>> is addressed to and may include private and confidential information that >>>> may not be disclosed, made public nor used in any way at all. Unauthorized >>>> use of the information in this electronic mail message may be subject to >>>> the penalties set forth by Chilean law. If you have received this >>>> electronic mail message in error, we ask you to destroy the message and >>>> its attached file(s) and to immediately notify the sender by answering >>>> this message. >>>> >>>> _______________________________________________ >>>> zeromq-dev mailing list >>>> [email protected] >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >> >> -- >> Gerry Steele >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
