I think it will be easy to tweak the libzmq code yourself to support this behavior. Try to do it with a new socket option and make a pull request. I can help you with finding the right place in the code.
On Mar 3, 2017 23:20, "Marcin Romaszewicz" <[email protected]> wrote: > Yeah, you need a way to throttle back the publisher. Have the many nodes > tell the publisher to slow down if they detect too high of a drop rate. > Your publisher can then decide whether to throttle for the slowest client, > or for the fastest client. Also, pub/sub may not be the right way to do > this for 1:many if the value of 'many' is very high, because each message > you send will be sent 1:1 to each subscriber, slowing them all down. Some > kind of tree based fanout may work better. > > -- Marcin > > On Fri, Mar 3, 2017 at 12:49 PM, Justin Karneges <[email protected]> > wrote: > >> And what if there are no subscribers? Publishing should block? >> >> It sounds like your goal is some kind of sensible flow control (I'm not >> sure what else an "at least 1 subscriber can read" use-case really is). My >> advice is to publish at a fixed rate, for example by sleeping between >> publishes. >> >> I do wish ZMQ_RATE worked for all PUB sockets and not just PGM-based. >> >> On Fri, Mar 3, 2017, at 12:31 PM, Francesco wrote: >> >> Hi all, >> >> I'm new to ZeroMQ but so far all my experiments with it have been quite >> positive, so thanks for your great work! >> >> However, one thing that I cannot really sort out is the following: I have >> written a small sample program where I create a ZMQ_PUB socket and I >> continuously zmq_msg_send() messages in it, never sleeping. This is >> simulating what I will do in a larger program where I plan to use ZeroMQ. >> Such program is a massively-parallel utility that needs to send in a fanout >> fashion (1 to many) several short messages per second (up to say 1million >> messages / second). >> >> In my scenario I don't care about subscribers joining late or eventually >> loosing some messages (if the network is slow or the subscriber itself is >> too slow). I do care however about detecting such conditions where messages >> are dropped. >> >> My problem is the "fast producer" one: this sample program shows that the >> zmq_msg_send() never returns an error; doing some math such my little >> utility says that is publishing data at rates up to 400Gbps... the only >> problem is that the NIC on the computer is a 1Gbps NIC. Of course all >> subscribers report missing 99% of messages (I put a sequence number in the >> messages I send). >> >> This problem does not appear to be new, some interesting references I >> found are: >> >> http://grokbase.com/t/zeromq/zeromq-dev/11ca11s9b9/pub-sub- >> pattern-rate-control-and-backpressure (Dec 2011) >> http://grokbase.com/t/zeromq/zeromq-dev/12797gy703/notify-s >> end-er-that-theyve-hit-the-high-water-mark (Jul 2012) >> http://grokbase.com/t/zeromq/zeromq-dev/129n2e2sx5/high-wat >> er-mark-notification-for-publisher (Sep 2012) >> >> I experimented a little bit with ZeroMQ and found that: >> - ZMQ_RATE is not doing anything >> - ZMQ_XPUB_NODROP set to 1 works BUT basically ties the publisher to >> the SLOWEST subscriber: from my experiments it looks like (despite the >> documentation) the publisher now blocks every time there is just 1 >> subscriber queue that has hit the HWM >> >> as written in some of these posts what I would like to have is a NODROP >> socket option that allows my publisher to send data as fast as the FASTEST >> subscriber can handle. >> >> So here's my question: is there any way to achieve the above, i.e. block >> the publisher if ALL subscriber queues hit their HWM ? >> >> >> Thanks a lot for any hint! >> >> Francesco >> >> >> >> >> >> *_______________________________________________* >> zeromq-dev mailing list >> [email protected] >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
