Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 18, 2017, at 23:28, Miroslav Lichvar wrote: > > On Wed, Jan 18, 2017 at 08:41:03PM -0800, Denny Page wrote: >> Tx -> Rx timestamp: >> Uncompensated3175ns (stddev 110ns) >> Compensated 3175ns >> >> Connection

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Jan 18, 2017 at 08:41:03PM -0800, Denny Page wrote: > Tx -> Rx timestamp: > Uncompensated3175ns (stddev 110ns) > Compensated 3175ns > > Connection types: ExpectedError > Regular switch 6721ns -3546ns >

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Jan 18, 2017 at 09:49:13AM -0800, Denny Page wrote: > And To be clear, this would not address phy/mac delay/asymmetry issues. It > would just replace the call to PTP_SYS_OFFSET. Instead of an array that you > have to interpolate, you just get a single (accurate) value. This would be >

Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 18, 2017, at 09:11, Denny Page wrote: > > >> On Jan 18, 2017, at 05:43, Miroslav Lichvar > > wrote: >> Denny, >> >> there is apparently a new ioctl for measuring the offset between the >> NIC clock and system clock,

Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 18, 2017, at 05:43, Miroslav Lichvar wrote: > Denny, > > there is apparently a new ioctl for measuring the offset between the > NIC clock and system clock, which is supported on some onboard NICs > (that share clock with the CPU?). It's called

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Jan 05, 2017 at 10:15:08AM -0800, Denny Page wrote: > Yes, comp is an abbreviation for compensation. One will always an add and the > other is always a subtract. This is because the timestamps are defined as > being at the php layer (last bit of the SFD), but the timestamps are being >

Re: [chrony-dev] SW/HW timestamping on Linux

That would be great. For option names, I would suggest something like: hwtimestamp igb0 txcomp 178 rxcomp 448 txcomp is the number of nanoseconds to add to tx timestamps rxcomp is the number of nanoseconds to subtract from rx timestamps Thanks, Denny > On Jan 02, 2017, at 05:47, Miroslav

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Dec 14, 2016 at 10:34:19PM -0800, Denny Page wrote: > Btw, it might be good for Chrony to support configuration for corrections > similar to what is discussed in that thread: > > delayAsymmetry - corrects for any asymmetry between the Rx and Tx paths > egressLatency - corrects the

Re: [chrony-dev] SW/HW timestamping on Linux

Btw, it might be good for Chrony to support configuration for corrections similar to what is discussed in that thread: delayAsymmetry - corrects for any asymmetry between the Rx and Tx paths egressLatency - corrects the transmit latency at the MAC/PHY ingressLatency - corrects the receive

Re: [chrony-dev] SW/HW timestamping on Linux

Interesting! I would not have thought of this. Both the i211 and i354 have the SDP pins. Not exactly a pre-made PPS input, but possible with some soldering and some code. Denny > On Dec 13, 2016, at 23:21, Miroslav Lichvar wrote: > > On Tue, Dec 13, 2016 at 05:03:15PM

Re: [chrony-dev] SW/HW timestamping on Linux

Also, would it be possible to allow hardware timestamps to be used on one interface, and software timestamps on another (without falling back to daemon timestamps)? Thanks, Denny -- To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" in the subject. For help email

Re: [chrony-dev] SW/HW timestamping on Linux

Unfortunately, I’ve not yet received any response on the Intel list. While it’s quite clear that both the i211 and i354 require some compensation, I don’t have a good way to determine it other than by shots in the dark. The i211 seems to correspond with the i210 correction parameters, so I am

Re: [chrony-dev] SW/HW timestamping on Linux

Thank you > On Apr 11, 2017, at 12:14, Miroslav Lichvar wrote: > > On Wed, Dec 07, 2016 at 11:26:19PM -0800, Denny Page wrote: >> Are you running linuxptp to manage the clock or letting it free-run? I have >> some ptp servers, but have been focused on testing ntp with

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Dec 07, 2016 at 11:26:19PM -0800, Denny Page wrote: > Are you running linuxptp to manage the clock or letting it free-run? I have > some ptp servers, but have been focused on testing ntp with chrony managing > the clock. The PHC is free running. As long as the system clock can stay

Re: [chrony-dev] SW/HW timestamping on Linux

Are you running linuxptp to manage the clock or letting it free-run? I have some ptp servers, but have been focused on testing ntp with chrony managing the clock. Thanks, Denny > On Apr 11, 2017, at 11:56, Miroslav Lichvar wrote: > > I've synchronised the system clock

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Dec 07, 2016 at 04:31:38PM -0800, Denny Page wrote: > > > On Dec 07, 2016, at 09:20, Miroslav Lichvar wrote: > > > > Yes, I think that might be helpful. I spent some time today comparing > > your method with the current code and at least on my system with i210 > > I

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Dec 05, 2016 at 02:35:22PM -0800, Denny Page wrote: > > > On Dec 05, 2016, at 01:05, Miroslav Lichvar wrote: > > > > If I understand your change correctly, it increases the minimum > > acceptable delay, which increases the number of combined readings and > > makes

Re: [chrony-dev] SW/HW timestamping on Linux

Here is an example of where I am at after the change: 210 Number of sources = 6 MS Name/IP address Stratum Poll Reach LastRx Last sample === ^* 192.168.230.240 1 0 377 0

Re: [chrony-dev] SW/HW timestamping on Linux

> On Dec 05, 2016, at 01:05, Miroslav Lichvar wrote: > > If I understand your change correctly, it increases the minimum > acceptable delay, which increases the number of combined readings and > makes the offset more stable. I'm worried on a busy system bad > readings could

Re: [chrony-dev] SW/HW timestamping on Linux

On Sun, Dec 04, 2016 at 10:56:57PM -0800, Denny Page wrote: > I’m still chasing the offset issue. However on the spikes/noise issue, the > patch below addresses the noise and low level spikes that I was seeing. I was > seeing errors of 2-60ns off in the average offset for the nic clock, which >

Re: [chrony-dev] SW/HW timestamping on Linux

I’m still chasing the offset issue. However on the spikes/noise issue, the patch below addresses the noise and low level spikes that I was seeing. I was seeing errors of 2-60ns off in the average offset for the nic clock, which this corrects. I’m still seeing the occasional larger spike, but

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 28, 2016 at 09:22:42AM -0800, Denny Page wrote: > Before I try and make a case to the driver and hardware folk, I think I need > to be able to explain how stamps on both two linux systems can sometimes be > in agreement with stamps on the second interface and sometimes not. Given >

Re: [chrony-dev] SW/HW timestamping on Linux

On Sun, Nov 27, 2016 at 01:06:20PM -0800, Denny Page wrote: > Here are the various test restuls: > > igb0 @ 1Gb; igb3 @ 100Mb direct connect: 192.168.230.245 shows offset of > +1230ns > igb0 @ 100Mb; igb3 @ 100Mb, direct connect: 192.168.230.245 shows no offset > igb0 @ 1Gb, igb3 @ 1Gb via

Re: [chrony-dev] SW/HW timestamping on Linux

I still have one significant offset mystery with the latest version. In the tests described below, the following IP addresses are involved: 192.168.230.2 Linux box. igb0 connected to main switch at 1Gb. 192.168.230.3 Linux box. igb0 connected to main switch, speed as described in tests.

Re: [chrony-dev] SW/HW timestamping on Linux

That cured my issue with offset between the hosts. > On Nov 24, 2016, at 06:57, Miroslav Lichvar wrote: > > On Thu, Nov 24, 2016 at 06:46:36AM -0800, Denny Page wrote: >> I do not have xleave enabled. Can you explain more as to why you would >> expect to see such an

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 24, 2016 at 06:46:36AM -0800, Denny Page wrote: > I do not have xleave enabled. Can you explain more as to why you would expect > to see such an offset without xleave? The offset is calculated from four timestamps. Two local and two remote. Without xleave the peer can't deliver the

Re: [chrony-dev] SW/HW timestamping on Linux

I do not have xleave enabled. Can you explain more as to why you would expect to see such an offset without xleave? Thanks, Denny > On Nov 24, 2016, at 05:11, Miroslav Lichvar wrote: > > That suggests the interleaved mode is not working. Are the peers specified > with

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Nov 23, 2016 at 03:24:56PM -0800, Denny Page wrote: > I am now seeing better standard deviations with hardware timestamping than > software timestamping. Thank you. > > Couple of caveats: > - I need to disable priority scheduling (-P). With priority scheduling, > software stamps still

Re: [chrony-dev] SW/HW timestamping on Linux

I am now seeing better standard deviations with hardware timestamping than software timestamping. Thank you. Couple of caveats: - I need to disable priority scheduling (-P). With priority scheduling, software stamps still have lower stddev. - I can only use a single ethernet interface. With

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, what version of the kernel are you using for testing? And what are you using for scheduling priority? Denny -- To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" in the subject. For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 22, 2016 at 09:06:08AM -0800, Denny Page wrote: > I tested with that earlier. Almost all the timestamps were rejected. But I > was using poll 0. I’ll have to test again with a longer polling interval. Rejected meaning measurements with 'D H' had a failing test bit in the log? The

Re: [chrony-dev] SW/HW timestamping on Linux

On Fri, Nov 18, 2016 at 03:37:07PM +0100, Miroslav Lichvar wrote: > On Thu, Nov 17, 2016 at 05:44:23PM -0800, Denny Page wrote: > > Although reduced, I’m still seeing spikes with the patch below. > > I'm not sure what could be wrong at this point. Maybe it really is a > kernel or HW issue. I'm

Re: [chrony-dev] SW/HW timestamping on Linux

I tested with that earlier. Almost all the timestamps were rejected. But I was using poll 0. I’ll have to test again with a longer polling interval. Denny > On Nov 22, 2016, at 07:43, Miroslav Lichvar wrote: > > Can you confirm that with the patch I sent you earlier >

Re: [chrony-dev] SW/HW timestamping on Linux

Perfect. Thank you. Denny > On Nov 22, 2016, at 07:34, Miroslav Lichvar wrote: > > I've pushed to the git an implementation of the idea I proposed > earlier, which assumes symmetric position of UDP data in received and > transmitted packets. You can modify the calculation

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 09:29:13PM -0800, Denny Page wrote: > Just to make sure I test correctly, would you mind creating a diff for a test > version for me? One that a can plug a hard coded number of nanoseconds > correction for just packet receive? I've pushed to the git an implementation of

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, Just to make sure I test correctly, would you mind creating a diff for a test version for me? One that a can plug a hard coded number of nanoseconds correction for just packet receive? Thanks, Denny > On Nov 21, 2016, at 17:26, Denny Page wrote: > > I may hard

Re: [chrony-dev] SW/HW timestamping on Linux

I would not have expected this, and I haven’t dug into it, but looking at the time to generate timestamps for pcap, I am seeing times of more than a second(!). I am wondering what happens if we get to the point of sending the next request before we have received the transmit timestamp for the

Re: [chrony-dev] SW/HW timestamping on Linux

I checked this. It appears to be completely symmetrical at the switch level. For both the 1Gb and 100Mb ports, the delta I see for NTP packets is a very consistent 912ns on a 1Gb mirror port. This is exactly as expected: preamble: 56 bits SFD: 8 bits MAC dest: 48 bits MAC src: 48 bits

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 11:25:10AM -0800, Denny Page wrote: > On Nov 21, 2016, at 10:59, Miroslav Lichvar wrote: > > So, in order to make this as simple as possible, we could make an > > assumption that received packets have the same format as transmitted > > packets. At

Re: [chrony-dev] SW/HW timestamping on Linux

> On Nov 21, 2016, at 10:59, Miroslav Lichvar wrote: > > On Mon, Nov 21, 2016 at 10:28:26AM -0800, Denny Page wrote: >> The only layer we know the size of is the UDP layer. The Ethernet layer can >> have VLAN tag. Yes, it’s only 32 bits, but error is error. The IP layer

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 10:28:26AM -0800, Denny Page wrote: > The only layer we know the size of is the UDP layer. The Ethernet layer can > have VLAN tag. Yes, it’s only 32 bits, but error is error. The IP layer may > have option headers. This is rare with IPv4, but not with IPv6. With TX

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, We actually don’t know the length of the data at layer 2. We can’t even guarantee the length at layer 3. Dispensing with layer numbers from now on. We have three levels we have to concern ourselves with: Ethernet IP UDP The only layer we know the size of is the UDP layer. The

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, This is called a cut-through, and is not the default. The reason it isn’t the default is because it propagates invalid frames (runts, run-ons, crc errors, etc.) beyond the source port. Store and forward is the default. Cut-though generally doesn’t work with port speed mismatch and

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 09:38:36AM -0800, Denny Page wrote: > Given host A on a 1Gb link and host B on a 100Mb link: > > A -> Switch transmission time 754ns > Switch -> B transmission time 7540ns > Total time 8294ns As I understand it, modern switches don't wait until they have whole

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, There is no correction necessary. The speed change itself does not have any impact because the transmission time is symmetrical. Given host A on a 1Gb link and host B on a 100Mb link: A -> Switch transmission time 754ns Switch -> B transmission time 7540ns Total time 8294ns

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I found the problem. The hardware receive timestamps are incorrect. It doesn’t matter if there is a switch involved or not. NTP depends upon round trip latency being symmetrical for it’s accuracy, both in local network and in remote networks. To help ensure this, NTP uses preamble

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I believe that the hardware NTP device, chrony, or both, are striking/calculating timestamps incorrectly. I have a test in mind that will allow me to determine if this is correct, and if so which. Back to you soon. Denny > On Nov 18, 2016, at 00:00, Miroslav Lichvar

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 17, 2016 at 05:44:23PM -0800, Denny Page wrote: > Although reduced, I’m still seeing spikes with the patch below. I'm not sure what could be wrong at this point. Maybe it really is a kernel or HW issue. I'm wondering what would be the best way to confirm or reject that idea. --

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 17, 2016 at 05:49:44PM -0800, Denny Page wrote: > This port speed differential appears to result in a asymmetry in > transmit/receive time which significantly affects the calculations. If I lock > the monitor host port at 100Mb, all three units show precise synchronization, > both

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I’ve found the issue with the 2.2us offset between a locally attached vs across the switch. It’s not the switch. The act of crossing the switch itself seems to be negligible. The problem appears to be one of asymmetry arising from port speed mismatch. The hardware NTP device uses

Re: [chrony-dev] SW/HW timestamping on Linux

Although reduced, I’m still seeing spikes with the patch below. Denny > On Nov 16, 2016, at 00:53, Miroslav Lichvar wrote: > > On Tue, Nov 15, 2016 at 08:24:37PM -0800, Denny Page wrote: >> With the latest drop in the repo, I’m still seeing the wild spikes in the >>

Re: [chrony-dev] SW/HW timestamping on Linux

To be clear, would I still expect to see ‘D H’ in the measurements log with this change? Thanks, Denny > On Nov 16, 2016, at 00:53, Miroslav Lichvar wrote: > > Hm, the fix helped with the spikes I was seeking. Did we rule out the > possibility that in your case the

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I’m sorry, I didn’t mean to imply that the spikes were an unrelated to the ordering issue. I do believe that the spikes are related to the hardware timestamp ordering issue. I’ll try to set up some of the tests you’ve requested later this evening, but it may take a day for me to

Re: [chrony-dev] SW/HW timestamping on Linux

Yes, all ports on the monitoring system are identical. The i354 is a 4 port chip, and all the ethernet ports on the monitoring unit are connected through that same chip. The general server has both I354 and I211 chips. I shut everything down on the server to conduct a test. It didn’t matter

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 15, 2016 at 09:39:40PM -0800, Denny Page wrote: > > While I can get my head around a differential of 250ns resulting from the > switch, I’m finding it very difficult to believe the almost 2500ns > differential that appears when hardware time stamping is enabled. > > Any thoughts

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 15, 2016 at 08:24:37PM -0800, Denny Page wrote: > With the latest drop in the repo, I’m still seeing the wild spikes in the > standard deviation with hardware time stamping against the fast responding > hardare units. I'm also still seeing a better base deviation using software >

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 15, 2016 at 08:14:28PM -0800, Denny Page wrote: > The chip is actually a I354, which is slightly different than the I350, but I > don’t think it matters much. I also have interfaces with I211 chips, and the > ordering issue appears to happen there as well. I don’t think the sleep

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, There is an additional issue, which is perhaps unrelated to the other issues discussed. Background: Each target IP is a hardware based NTP server. IPs 240 and 244 are across the switch. Chrony is synchronizing with these two IPs. Incremental latency from the switch is approximately

Re: [chrony-dev] SW/HW timestamping on Linux

With the latest drop in the repo, I’m still seeing the wild spikes in the standard deviation with hardware time stamping against the fast responding hardare units. I'm also still seeing a better base deviation using software timestamps against them as well. I do see better results with

Re: [chrony-dev] SW/HW timestamping on Linux

The chip is actually a I354, which is slightly different than the I350, but I don’t think it matters much. I also have interfaces with I211 chips, and the ordering issue appears to happen there as well. I don’t think the sleep after the send is going to affect the order of the timestamp and

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 14, 2016 at 03:40:32PM +0100, Miroslav Lichvar wrote: > On Sat, Nov 12, 2016 at 10:36:55AM -0800, Denny Page wrote: > > Here is a reasonable visual representation of what I am seeing. The section > > on the left (before 8:00) is with hardware timestamping, while the section > > on

Re: [chrony-dev] SW/HW timestamping on Linux

Setting noselect for all sources seems to have no impact on standard deviation. Disabling all forms of dynamic tic (CONFIG_HZ_PERIODIC=y) seems to have some effect in reducing the number of “D H”, but does not appear to have much of an impact on the standard deviation. I am returning

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 14, 2016 at 10:47:52AM -0800, Denny Page wrote: > I’m not sure I understand what effect a delay in NIO_Linux_RequestTxTimestamp > would have. That I see, NIO_Linux_RequestTxTimestamp builds a control message > structure but does not make any level 2 calls. Introducing a delay here >

Re: [chrony-dev] SW/HW timestamping on Linux

Yes, I these on the monitoring system. For all servers/peers. Denny > On Nov 14, 2016, at 05:35, Miroslav Lichvar wrote: > > Do you see in measurements.log any entries with 'D K' and > '111 111 ' in the columns with tests results (i.e. were any 'D K' > measurements

Re: [chrony-dev] SW/HW timestamping on Linux

On Sat, Nov 12, 2016 at 10:36:55AM -0800, Denny Page wrote: > Here is a reasonable visual representation of what I am seeing. The section > on the left (before 8:00) is with hardware timestamping, while the section on > the right is with software timestamping. maxdelaydevratio of 4 in both

Re: [chrony-dev] SW/HW timestamping on Linux

On Sat, Nov 12, 2016 at 10:36:55AM -0800, Denny Page wrote: > Here is a reasonable visual representation of what I am seeing. The section > on the left (before 8:00) is with hardware timestamping, while the section on > the right is with software timestamping. maxdelaydevratio of 4 in both

Re: [chrony-dev] SW/HW timestamping on Linux

Here is a reasonable visual representation of what I am seeing. The section on the left (before 8:00) is with hardware timestamping, while the section on the right is with software timestamping. maxdelaydevratio of 4 in both cases. I think I have a kernel/driver problem. Denny

Re: [chrony-dev] SW/HW timestamping on Linux

The average offset is ~300ns. I would not expect there to be so much asymmetry with the switch, but I suppose anything is possible. If I enable hardware time stamps the offset jumps to ~2100ns. Denny > On Nov 11, 2016, at 02:22, Miroslav Lichvar wrote: > > This looks

Re: [chrony-dev] SW/HW timestamping on Linux

> On Nov 11, 2016, at 02:17, Miroslav Lichvar wrote: > > The mixed results in tests 3, 4 and 8 are not as expected, however. > Can you please try running one of those tests with 'acquisitionport > 1' in chrony.conf and post the debug output (after ./configure >

Re: [chrony-dev] SW/HW timestamping on Linux

Turns out that after I run it for longer period while, I see mixed D H and H H for test 5 as well. > On Nov 10, 2016, at 13:35, Denny Page wrote: > > test 5 > - hw stamps on for igb0 > - ntp servers attached via igb0 enabled, server attached via igb3 disabled > - result: H

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 10, 2016 at 02:38:04PM -0800, Denny Page wrote: > Things are working quite well with software timestamping. > > This is from my monitoring system: > 210 Number of sources = 6 > Name/IP AddressNP NR Span Frequency Freq Skew Offset Std Dev >

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 10, 2016 at 01:35:22PM -0800, Denny Page wrote: > Here is the test setup: 3 identical hardware NTP units. Two accessible via > switch on igb0. One accessible vi igb3 as a directly connected IP point to > point. Here are test results: Thanks for running the tests, much appreciated.

Re: [chrony-dev] SW/HW timestamping on Linux

Things are working quite well with software timestamping. This is from my monitoring system: 210 Number of sources = 6 MS Name/IP address Stratum Poll Reach LastRx Last sample === ^?

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, The kernel is 4.4.26. The problem appears to be hardware timestamping and multiple interfaces. Here is the test setup: 3 identical hardware NTP units. Two accessible via switch on igb0. One accessible vi igb3 as a directly connected IP point to point. Here are test results: test 1

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 10, 2016 at 11:25:31AM -0800, Denny Page wrote: > Miroslav, > > And D H? D means daemon, i.e. no SW or HW timestamp. The first one is for transmit timestampt, the other one for receive timestamp. If only the first entry in the log is like that, then it's ok. If not, something is