I would have use for some of this stuff as well. However, I don't think it really feels at home in zmq itself ? What about a middleware layer between zmq and user apps that takes care of this ?
Kr, Diego Duclos On Thu, Jul 18, 2013 at 8:47 PM, CFK <[email protected]> wrote: > Thank you Peter, that was what I was afraid of. Stephen, I agree with you > that having an option for dropping messages would be useful. So, as a > strawman proposal, here are the options I'd like to see added to ZMQ: > > ZMQ_EXPIRE_TIME_DELTA - If the time between when the user's call to send() > and the time the message is going to be sent is greater than this amount, > then the message is silently discarded. This is useful for messages that > are useless if old. > > ZMQ_PROBABILISTIC_HWM - Similar to the high water mark, except that it > isn't a hard limit. When a message is up for transmission, ZMQ may choose > to drop the message. The probability that a message will be dropped is > length(ZMQ output buffer)/ZMQ_PROBABILISTIC_HWM. So the more messages > there are in the buffer, the greater the probability that the message at > the head of the queue will be dropped. This allows back pressure to be > monitored by user applications. For applications that can scale how much > data they transmit this can be VERY useful. For example, I have several > options on how to compress aggregate/compress data. Compression works > better on larger messages, but because I'm time sensitive, I want to get > messages out quickly. By monitoring the back pressure, I can find the > sweet spot for my application, even as it changes. Which brings up the > following options: > > ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW and ZMQ_TRANSMISSION_SUCCESS_RATE - > ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW is an option on a socket. ZMQ does a > moving average over the last ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW messages > that the user tried to send to calculate the fraction that were actually > transmitted (NOT received! This is just on the sender's side, seeing what > ZMQ sent out). You can monitor this via > zmq_socket_monitor<http://api.zeromq.org/3-2:zmq-socket-monitor>and > ZMQ_TRANSMISSION_SUCCESS_RATE. > > Thoughts or comments? > > Thanks, > Cem Karan > > > > On Thu, Jul 18, 2013 at 1:44 PM, Pieter Hintjens <[email protected]>wrote: > >> There is no option to drop old messages. >> >> Pieter >> On Jul 18, 2013 4:45 PM, "CFK" <[email protected]> wrote: >> >>> Hi all, I'm trying to figure out what socket options I need to set under >>> ZMQ 3.2.3 to force ZMQ XPUB sockets to drop messages if they are old. I >>> have time-sensitive messages, any/all of which can be dropped if they are >>> older than some amount (set by the option) before ZMQ starts actual >>> transmission. Is there an option for this? From my testing, ZMQ_SNDTIMEO >>> does not seem to do this, and none of the other options I've read about >>> seem to be applicable. As it stands, I'm using pyzmq's MessageTracker >>> interface, along with a hackish thread interface, to queue all messages I >>> plan on sending until the currently sent message is actually transmitted, >>> before starting on the next message. This is suboptimal, and I'd like to >>> do something better. >>> >>> Note that any solution MUST work with XPUB/XSUB as they operate over UDP >>> (via PGM). I'm operating on a lossy wireless network, and cannot tolerate >>> how long TCP takes to timeout a connection, or how long it takes to figure >>> out a connection is back up. >>> >>> Thanks, >>> Cem Karan >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
