Hi Bill, > At the risk of telling you something you already know, have you thought about how these timestamps are being generated? I assume that the multiple PUBs are, or at least can be, located on different machines. In that scenario the problem of keeping clocks in sync between the machines becomes non-trivial.
Yes, you're right the producers will be (not always) on different servers. These microservices are actually running in a Kubernetes cluster and we ensure that all the worker nodes have a valid NTP configuration (typically referring to stratum 1 clock sources)... I know that the time sync across the servers will be mostly likely in the 500 usec-5msec range.... and honestly when we are in msec-range, the accuracy is very bad considering that the microservices exchange up to 300-500 kmsg/sec, taking about 1Gbps of bandwidth for each link... However, that's the best we have currently. > The best explanation I’ve seen is this: https://queue.acm.org/detail.cfm?id=2878574, which in turn references Lamport’s seminal work: http://lamport.azurewebsites.net/pubs/time-clocks.pdf. Thanks - I read the first one about NTP and PTP and it's a good basic introduction to these 2 protocols. On the same page I suggest another not-so-technical reading: https://www.newyorker.com/tech/annals-of-technology/the-thorny-problem-of-keeping-the-internets-time That article goes over the history of NTP and its creator and long-time maintainer... Thanks! Francesco Il giorno gio 10 nov 2022 alle ore 15:04 Bill Torpey <[email protected]> ha scritto: > Hi Francesco: > > At the risk of telling you something you already know, have you thought > about how these timestamps are being generated? I assume that the multiple > PUBs are, or at least can be, located on different machines. In that > scenario the problem of keeping clocks in sync between the machines becomes > non-trivial. > > The best explanation I’ve seen is this: > https://queue.acm.org/detail.cfm?id=2878574, which in turn references > Lamport’s seminal work: > http://lamport.azurewebsites.net/pubs/time-clocks.pdf. > > Hope this helps. > > Bill Torpey > > > On Nov 9, 2022, at 5:19 PM, Francesco <[email protected]> > wrote: > > Hi all, > > I have written two applications using ZMQ PUB-SUB pattern (over TCP > transport). > The subscriber application has its SUB socket connected to multiple PUBs > (multiple tcp endpoints). Each message sent by the PUB encodes the > timestamp (as obtained from clock_gettime() syscall at TX side using > monotonically increasing clock) of the event described by the ZMQ message. > > The subscriber needs to process the data stream _strictly_ in order. > However the multiple publishers have no coordination and they will emit > messages at different rates, each with its own timestamp. The only > guarantee that I have, according to ZMQ docs, is that the SUB socket will > perform "fair dequeueing", but that's not enough to guarantee that every > zmq_msg_t received from the SUB socket will have a monotonically increasing > timestamp: it depends on the filling level of the TCP rx/tx kernel buffers, > the zmq HWMs, etc. > > For this reason I'm looking for some algorithm that > * allows me to push zmq_msg_t pulled out of the SUB socket (without strict > time ordering) > * allows me to pull out zmq_msg_t that have a timestamp monotonically > increasing > * introduces a fixed max latency of N msecs (configurable) > > Do you have any pointer for such kind of problem? > Anybody already hit a similar issue? > > Thanks for any help, > > Francesco Montorsi > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
