Dumb question, but why not a simple PUSH/PULL pattern and no queues? Each consumer PULLs one message to process. The process pushing data (which may be more than one) will block when the HWM is hit (all consumers are busy, and the “queue” from the producer is full). No need for “hold-on-i’m-busy” messages, as PUSH-PULL does it already by themselves.
On Jan 15, 2014, at 16:29, Chris Whiten <[email protected]> wrote: > I'm using zeromq in a competing consumer setup, where one process is pushing > data to a set of worker processes to parallelize some data processing. > > The publisher publishes data much more quickly than the workers can process > it, so I want the publisher to block when all of the consumers are too busy. > I'll decide if they're too busy by having them pull data from zeromq and push > them to an in-memory queue on each of the workers, and if that in-memory > queue is too large we temporarily stop accepting messages from zeromq. > > When all of the consumers are too busy, I wish for the client that is pushing > data to the consumers to block, so we don't lose large chunks of data. To > solve that, I've set up the extended request-reply pattern (as outlined in > Figure 16 of the zeromq guide at http://zguide.zeromq.org/page:all). This > seems to work in the happy case, but if a worker goes down the client will > never receive a response back from that worker, and it will hang > indefinitely. Have I selected the wrong pattern, or is there an easy way to > remedy this problem? > > My code to test this is at http://pastebin.com/CaZpVayu > > Thanks for your help > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
