Hi Randall, I never got around to building a reliable pipeline pattern, but I think it's pretty straight forward.
The general model is you send out X pieces of work to W workers, who send replies back to a sink. The potential failure is a worker dying and losing one or more messages. The recovery is to resend the entire batch. If the failure is rare enough, this is a sensible approach. If the failure is common, you probably want to switch to a more synchronous request-reply model like Majordomo. -Pieter On Sat, Jun 29, 2013 at 11:37 PM, Randall Nortman <[email protected]> wrote: > My apologies if this is somewhere in the guide and I've missed it. > Most of the "reliable" patterns in the guide seem to be about > request/response patterns. My desired behavior is: > > - A producer binds a socket > > - A consumer connects to the producer (one consumer, one producer) > > - The producer queues up to N messages for a consumer. As long as the > queue has not reached N, messages are only removed from the queue > once the producer is certain that the consumer has received the > message. Re-delivery of the same message is acceptable. Once the > queue reaches N messages, the producer discards the OLDEST message, > so that always the most recent N messages are queued. When the > consumer comes back, the N most recent messages are transmitted > (with confirmation of receipt before discarding, subject to queue > size, as usual). > > Based on my reading of the guide, ZMQ doesn't have anything out of the > box that looks like this. ZMQ's buffers and HWM simply don't work > that way; if anything, they make it slightly harder to reliably do > what I need to do. The closest I seem to be able to come up with is a > PUSH with the high water mark set to 1 and configured for non-blocking > sends. If a send would block, then I instead put the message in a > circular queue that I maintain myself. And I am pretty sure that even > that solution may occasionally drop a message when the consumer > disconnects, because the producer won't notice right away, and > whatever message was in flight at the time will be lost. > > Beyond that, it seems like I need to use one of the reliable > request/response patterns, using the responses as confirmation of > receipt (upon which I remove the confirmed message from my queue). It > would be nice to reduce network traffic by allowing the consumer to > confirm multiple messages in one response (rather than one > confirmation for every message). That's obviously no longer simple > REQ/REP, but probably something involving more than one socket on each > end. > > It seems like the only benefit I get from ZMQ if I want this kind of > behavior is that it handles reconnecting automatically. I'm kind of > feeling like I might as well use raw UDP, because I'm going to have to > do everything myself anyway. (I am not knocking ZMQ. I'm using it > for lots of other things that it's well-suited for. It just seems > this isn't one of those things.) > > Am I missing an easy way to do it? > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
