Hi, First I would like to thank the developers for a great project. While ZeroMQ does not solve *every* communication problem, it is certainly very well designed, very effective, and easy to use for the space of problems it can solve well. After working with it for a few weeks, some of the reasons for certain (what I initially thought unusual) design decisions become clear.
However, I encountered the following PUB/SUB pattern which did not so neatly into the patterns described in the guide. For instance, (just like in the guide), I want to build a replicated key value store. However, I want to be able to provide relatively strong replica consistency. For simplicity, say there is only 1 master where the contents are replicated onto a set of slaves. This can be implicated easily as a PUB/SUB system, and the Suicidal Snail pattern is effective for dealing with slow slaves/subscribers. However, if the size of the store is large (many GBs), killing the subscriber, or asking for a fresh snapshot, is very costly. In this case, the right thing to do (just as in a database system), will be to put back pressure and slow down writes on the master, so that all the slaves can keep up. I therefore would like a PUB socket which blocks on hwm instead of dropping messages. (Alternate designs include using ROUTER on the master and DEALER on the slave, and doing manual fanout. However, ROUTER also drops messages on hwm. Several alternate designs are also possible by manually designing sequence Acks from slaves to master, but that adds substantial update caching complexity on the master.) I therefore made a relatively simple patch as attached to add an additional socket option ZMQ_PUB_NODROP ( and equivalently ZMQ_XPUB_NODROP ). and have xsend() in xpub return EAGAIN if not all the pipes matching the message are ready for writing. Initial testing suggests that it seems to work fine, allowing all subscribers to receive every message regardless of the hwm. A limitation is that I cannot use zmq_poll to test if the PUB socket is writeable. (A conservative option is to have the PUB socket be not writeable unless all the connected SUB sockets are writeable). Yucheng
pub_no_drop.diff
Description: Binary data
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
