Thank you Peter, that was what I was afraid of. Stephen, I agree with you that having an option for dropping messages would be useful. So, as a strawman proposal, here are the options I'd like to see added to ZMQ:
ZMQ_EXPIRE_TIME_DELTA - If the time between when the user's call to send() and the time the message is going to be sent is greater than this amount, then the message is silently discarded. This is useful for messages that are useless if old. ZMQ_PROBABILISTIC_HWM - Similar to the high water mark, except that it isn't a hard limit. When a message is up for transmission, ZMQ may choose to drop the message. The probability that a message will be dropped is length(ZMQ output buffer)/ZMQ_PROBABILISTIC_HWM. So the more messages there are in the buffer, the greater the probability that the message at the head of the queue will be dropped. This allows back pressure to be monitored by user applications. For applications that can scale how much data they transmit this can be VERY useful. For example, I have several options on how to compress aggregate/compress data. Compression works better on larger messages, but because I'm time sensitive, I want to get messages out quickly. By monitoring the back pressure, I can find the sweet spot for my application, even as it changes. Which brings up the following options: ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW and ZMQ_TRANSMISSION_SUCCESS_RATE - ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW is an option on a socket. ZMQ does a moving average over the last ZMQ_TRANSMISSION_SUCCESS_RATE_WINDOW messages that the user tried to send to calculate the fraction that were actually transmitted (NOT received! This is just on the sender's side, seeing what ZMQ sent out). You can monitor this via zmq_socket_monitor<http://api.zeromq.org/3-2:zmq-socket-monitor>and ZMQ_TRANSMISSION_SUCCESS_RATE. Thoughts or comments? Thanks, Cem Karan On Thu, Jul 18, 2013 at 1:44 PM, Pieter Hintjens <[email protected]> wrote: > There is no option to drop old messages. > > Pieter > On Jul 18, 2013 4:45 PM, "CFK" <[email protected]> wrote: > >> Hi all, I'm trying to figure out what socket options I need to set under >> ZMQ 3.2.3 to force ZMQ XPUB sockets to drop messages if they are old. I >> have time-sensitive messages, any/all of which can be dropped if they are >> older than some amount (set by the option) before ZMQ starts actual >> transmission. Is there an option for this? From my testing, ZMQ_SNDTIMEO >> does not seem to do this, and none of the other options I've read about >> seem to be applicable. As it stands, I'm using pyzmq's MessageTracker >> interface, along with a hackish thread interface, to queue all messages I >> plan on sending until the currently sent message is actually transmitted, >> before starting on the next message. This is suboptimal, and I'd like to >> do something better. >> >> Note that any solution MUST work with XPUB/XSUB as they operate over UDP >> (via PGM). I'm operating on a lossy wireless network, and cannot tolerate >> how long TCP takes to timeout a connection, or how long it takes to figure >> out a connection is back up. >> >> Thanks, >> Cem Karan >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
