Hi Francesco: At the risk of telling you something you already know, have you thought about how these timestamps are being generated? I assume that the multiple PUBs are, or at least can be, located on different machines. In that scenario the problem of keeping clocks in sync between the machines becomes non-trivial.
The best explanation I’ve seen is this: https://queue.acm.org/detail.cfm?id=2878574 <https://queue.acm.org/detail.cfm?id=2878574>, which in turn references Lamport’s seminal work: http://lamport.azurewebsites.net/pubs/time-clocks.pdf <http://lamport.azurewebsites.net/pubs/time-clocks.pdf>. Hope this helps. Bill Torpey > On Nov 9, 2022, at 5:19 PM, Francesco <[email protected]> wrote: > > Hi all, > > I have written two applications using ZMQ PUB-SUB pattern (over TCP > transport). > The subscriber application has its SUB socket connected to multiple PUBs > (multiple tcp endpoints). Each message sent by the PUB encodes the timestamp > (as obtained from clock_gettime() syscall at TX side using monotonically > increasing clock) of the event described by the ZMQ message. > > The subscriber needs to process the data stream _strictly_ in order. However > the multiple publishers have no coordination and they will emit messages at > different rates, each with its own timestamp. The only guarantee that I have, > according to ZMQ docs, is that the SUB socket will perform "fair dequeueing", > but that's not enough to guarantee that every zmq_msg_t received from the SUB > socket will have a monotonically increasing timestamp: it depends on the > filling level of the TCP rx/tx kernel buffers, the zmq HWMs, etc. > > For this reason I'm looking for some algorithm that > * allows me to push zmq_msg_t pulled out of the SUB socket (without strict > time ordering) > * allows me to pull out zmq_msg_t that have a timestamp monotonically > increasing > * introduces a fixed max latency of N msecs (configurable) > > Do you have any pointer for such kind of problem? > Anybody already hit a similar issue? > > Thanks for any help, > > Francesco Montorsi > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
