If you block the pub when the sub hits their HWM, then it wouldn't be pub-sub anymore. The one positive property of pub-sub is that pub side never needs to worry about the subs. This allows you to scale... to a lot of subs. If you start placing at-least-once requirements on this, I suggest you don't use pub-sub, but instead design your own protocol. For example, all the queues like rabbitmq do this. I think this is what you actually want - because sounds like you're not willing to loose messages. At some point you *have* to decide to shed some load, and I find this article very nice: http://ferd.ca/queues-don-t-fix-overload.html
On Sun, 5 Mar 2017 at 07:03 Francesco <[email protected]> wrote: > Hi Justin, > below my replies: > > 2017-03-03 21:49 GMT+01:00 Justin Karneges <[email protected]>: > > And what if there are no subscribers? Publishing should block? > IMO in such case the publisher is free to publish messages at any rate > (in my case, 400Gbps!!)... after all... who cares if there are no > subscribers? :) > > > > It sounds like your goal is some kind of sensible flow control (I'm not > sure > > what else an "at least 1 subscriber can read" use-case really is). > Yes exactly: the problem is that the only type of "flow control" that > can be done now (ZMQ_XPUB_NODROP=1) leads to a scenario that does not > scale well: if I have 1k subscribers and suddenly just one of them > slows down to 1message/sec for whatever reason (networking issue, > software bug, whatever), the publisher would slow down to 1message/sec > basically propagating the failure of 1 subscriber to the whole > network. > > Of course also the opposite case exists: if I have 1k subscribers and > suddenly just one of them increases its processing speed by a 10x > factor, the publisher would follow it publishing 10x more messages per > sec potentially causing other subscribers to drop lots of messages. > However I think this case is much more unlikely than the first one. > > > > My advice > > is to publish at a fixed rate, for example by sleeping between publishes. > > Unfortunately this is not a solution in my case, as the events I > publish are generated by other incoming events that I cannot control > and in general I will not know at which rate they arrive. > > Thanks, > Francesco > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
