Miroslav,

I found the problem. The hardware receive timestamps are incorrect. It doesn’t 
matter if there is a switch involved or not.

NTP depends upon round trip latency being symmetrical for it’s accuracy, both 
in local network and in remote networks. To help ensure this, NTP uses preamble 
timestamps for transmit and trailer timestamps for receive. The following 
article makes for good background reading:

      https://www.eecis.udel.edu/~mills/stamp.html

From that article:

        • A preamble timestamp is struck as near to the start of the packet as 
possible. The preferred point follows the last bit of the preamble and 
start-of-frame (STF) octet and before the first octet of the data.
        • A trailer timestamp is struck as near to the end of the packet as 
possible. On transmit this follows the last octet of the data and before the 
frame check sequence (FCS); on receive this follows the last octet of the FCS.

In the current Chrony implementation, we have the following timestamp types 
available:

        • Daemon timestamps. Time stamps generated in application space.
        • Software timestamps. Timestamps created by the kernel / driver using 
SOF_TIMESTAMPING_TX_SOFTWARE and SOF_TIMESTAMPING_RX_SOFTWARE. This is called 
“driver timestamping” in the article.
        • Hardware timestamps. Timestamps created by the network interface 
using SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_RX_HARDWARE.

With the current implementation, software timestamps are the most accurate. 
Depending on network speed, processor speed and load, daemon timestamps are 
second, and hardware timestamps third.


Looking at the transmit side, these are the steps involved:

        send() invoked
        … variance from time in kernel ...
        kernel invokes driver
        … variance from time in driver ...
        driver writes packet to network controller
        … variance from time in network controller (packet 
scheduling/buffering, inter-packet gap, preamble collisions, etc.) ...
        network controller sends SFD (start frame delimiter, called STF in the 
article)
        network controller sends data bytes
        network controller sends FCS (frame check sequence)

With daemon timestamps, Chrony generates the timestamp right before invoking 
send(). We suffer variance from time in kernel, network driver, and network 
controller. Software timestamping improves this by moving timestamp generation 
to the point immediately prior to the network driver writing the packet to the 
network controller, which removes the variance from the kernel and driver. 
Hardware timestamps improve this even further by moving timestamp generation to 
the point immediately following the SFD and prior to the data being sent, 
removing the last of the variance. The perfect preamble timestamp.


Looking at the receive side, these are the steps involved:

        network controller receives SFD
        network controller receives data bytes
        network controller receives FCS
        network controller generates interrupt
        … variance from time waiting to process interrupt ...
        driver invoked
        … variance from time in driver ...
        driver hands packet to kernel
        … variance from time in kernel ...
        select() returns
        recv() invoked

With daemon timestamps, Chrony generates the timestamp immediately after select 
returns. We suffer variance from time in the network driver and kernel. 
Software timestamping improves this by moving timestamp generation to the point 
immediately after the driver hands the packet to the kernel, which removes the 
vast majority of the variance from time spent in the kernel. Now we come to 
hardware timestamps. What we would desire from hardware timestamping is that it 
be generated immediately following the FCS, which would remove the rest of the 
variance and give us the perfect trailer timestamp. However that’s not how the 
network controller generates hardware receive timestamps. With network 
controllers, the hardware receive timestamps are generated immediately 
following the SFD, just as they are for transmit. See Figure 7-27 (Time Stamp 
Point) on page 454 here: 
http://www.intel.com/content/www/us/en/embedded/products/networking/ethernet-controller-i350-datasheet.html.

What this means for NTP is that hardware timestamps are off by a minimum of 752 
transmission bits with IPv4. This assumes no VLAN, and no IP option headers. In 
a 100Mb network, this means a guaranteed minimum timestamp error of 7.52 
microseconds.

In order to generate a correct receive timestamp from the Ethernet hardware 
timestamp, one needs to need to have the FCS timestamp, the current interface 
speed, and the length of the packet at the Ethernet level. This is doable, but 
requires use of raw sockets and is quite a bit of work. A simpler (and safer) 
approach would be to use a combination of hardware timestamping for send 
(SOF_TIMESTAMPING_TX_HARDWARE), and software timestamping for receive 
(SOF_TIMESTAMPING_RX_SOFTWARE). This is probably the best available option.

Denny


--
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Reply via email to