That was my original thought, and what I had tried before being pointed to the pattern I originally described. I've tried setting the high watermark, but this seems to just dictate how many messages the consumer should buffer before throwing messages away. I haven't been able to find any options to coax the publisher into blocking when this limit is hit.
Here is my push/pull pattern code that I had tried: http://pastebin.com/CJLaY5zk On Wed, Jan 15, 2014 at 12:43 PM, Bruno D. Rodrigues < [email protected]> wrote: > Dumb question, but why not a simple PUSH/PULL pattern and no queues? Each > consumer PULLs one message to process. The process pushing data (which may > be more than one) will block when the HWM is hit (all consumers are busy, > and the “queue” from the producer is full). No need for “hold-on-i’m-busy” > messages, as PUSH-PULL does it already by themselves. > > > On Jan 15, 2014, at 16:29, Chris Whiten <[email protected]> wrote: > > I'm using zeromq in a competing consumer setup, where one process is > pushing data to a set of worker processes to parallelize some data > processing. > > The publisher publishes data much more quickly than the workers can > process it, so I want the publisher to block when all of the consumers are > too busy. I'll decide if they're too busy by having them pull data from > zeromq and push them to an in-memory queue on each of the workers, and if > that in-memory queue is too large we temporarily stop accepting messages > from zeromq. > > When all of the consumers are too busy, I wish for the client that is > pushing data to the consumers to block, so we don't lose large chunks of > data. To solve that, I've set up the extended request-reply pattern (as > outlined in Figure 16 of the zeromq guide at > http://zguide.zeromq.org/page:all). This seems to work in the happy > case, but if a worker goes down the client will never receive a response > back from that worker, and it will hang indefinitely. Have I selected the > wrong pattern, or is there an easy way to remedy this problem? > > My code to test this is at http://pastebin.com/CaZpVayu > > Thanks for your help > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
