Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hmm that is a lot of messages to try to retain the order of. Your buffer time will need to be larger than the clock drift between publishers. Even you manage to keep your clocks within 1 second accuracy and use a 1 second buffer, that's still a million buffered packets. Maybe you can shard

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hi Justin, you're absolutely right, I want to do something very very similar to a jitter buffer. I 've found some implementations: https://github.com/icefreedom/jitter_buffer/tree/master/include/jitter_buffer https://github.com/alpartis/rtp.jitter but honestly they are so much focused on RTP that

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hi, It sounds like you need logic similar to a jitter buffer, which is commonly used in RTP media streaming (voip / TV airwaves). Basically, you have a time buffer (say 300 milliseconds) that timestamped messages are held in before processing. Each message sits in this queue for the defined time

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hi Francesco, Thank you so much. Trying that now. Is this set only on the publish socket? From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf Of Francesco Sent: Wednesday, December 6, 2017 08:26 To: ZeroMQ development list Subject: Re:

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hi Mark, have a look at ZMQ_XPUB_NODROP socket option. Despite the name, it applies also to PUB sockets (not only XPUB) and allows you to have a PUB/SUB without drops. I'm using that in my applications all the times. Btw I'm sending Mpps through PUB/SUB ZMQ sockets, so 100kpps should not be a big

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hi All, Slightly on and off topic at the same time: I am considering using PUB/SUB for distributing raw data packets from our instrument. While it is not absolutely critical that the packets be perfectly ordered, it is pretty important none are dropped. Our rate is fairly fast. I need to be

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

On Wed, 2017-12-06 at 14:38 +0100, Francesco wrote: > Hi Luca, > yes sorry: I mean in the case of multiple publishers. In such case > ZMQ will > apply fair-dequeueing of the per-publisher queues, right? Yes, that's correct. > The messages I publish have a timestamp inside them and I would like >

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hi Luca, yes sorry: I mean in the case of multiple publishers. In such case ZMQ will apply fair-dequeueing of the per-publisher queues, right? The messages I publish have a timestamp inside them and I would like to process, inside my SUBscriber the messages ordered by timestamp, but of course ZMQ

Re: [zeromq-dev] PUB/SUB model and time-ordering SUB-side

On Wed, 2017-12-06 at 12:55 +0100, Francesco wrote: > Hi all, > As you know when using PUB/SUB model the order of arrival of messages > at > SUB-side cannot be garantueed and generally speaking, the SUB will > receive > messages out of order. This is a well known problem, see e.g.: >  

[zeromq-dev] PUB/SUB model and time-ordering SUB-side

Hi all, As you know when using PUB/SUB model the order of arrival of messages at SUB-side cannot be garantueed and generally speaking, the SUB will receive messages out of order. This is a well known problem, see e.g.: https://cloud.google.com/pubsub/docs/ordering My question is: in case in the