Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 18, 2017, at 23:28, Miroslav Lichvar wrote: > > On Wed, Jan 18, 2017 at 08:41:03PM -0800, Denny Page wrote: >> Tx -> Rx timestamp: >> Uncompensated3175ns (stddev 110ns) >> Compensated 3175ns >> >> Connection types: Expected

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Jan 18, 2017 at 06:56:08PM +0100, Miroslav Lichvar wrote: > Avoiding the asymmetry on the PCIe bus is what I'm most interested in. > I'll add support for the new ioctl, but I wanted to test it first. I > think I found a system that has art and e1000e in our lab, so I'll see > if I can test

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Jan 18, 2017 at 08:41:03PM -0800, Denny Page wrote: > Tx -> Rx timestamp: > Uncompensated3175ns (stddev 110ns) > Compensated 3175ns > > Connection types: ExpectedError > Regular switch 6721ns -3546ns > Cu

Re: [chrony-dev] SW/HW timestamping on Linux

I’ve pushed a new version of ethtscal that greatly improves the accuracy/stability of results. With runs of 5 packets, I’m seeing consistency of +-5ns on loopback. If you pulled an older version, I recommend pulling a new one. Here is a sample run with i211 chips with loopback cable: Iter

Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 18, 2017, at 09:56, Miroslav Lichvar wrote: > > Avoiding the asymmetry on the PCIe bus is what I'm most interested in. I’m not sure how much asymmetry on the PICe bus there is. Or how it affects PTP_SYS_OFFSET. PCIe asymmetry/delay won't affect the actual timestamps. To my knowledge

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Jan 18, 2017 at 09:49:13AM -0800, Denny Page wrote: > And To be clear, this would not address phy/mac delay/asymmetry issues. It > would just replace the call to PTP_SYS_OFFSET. Instead of an array that you > have to interpolate, you just get a single (accurate) value. This would be > ve

Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 18, 2017, at 09:11, Denny Page wrote: > > >> On Jan 18, 2017, at 05:43, Miroslav Lichvar > > wrote: >> Denny, >> >> there is apparently a new ioctl for measuring the offset between the >> NIC clock and system clock, which is supported on some onboard NICs >

Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 18, 2017, at 05:43, Miroslav Lichvar wrote: > Denny, > > there is apparently a new ioctl for measuring the offset between the > NIC clock and system clock, which is supported on some onboard NICs > (that share clock with the CPU?). It's called PTP_SYS_OFFSET_PRECISE > and if I understan

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Dec 07, 2016 at 06:20:48PM +0100, Miroslav Lichvar wrote: > Yes, I think that might be helpful. I spent some time today comparing > your method with the current code and at least on my system with i210 > I see a shift in the distribution of the offset to one side when the > network is (heav

Re: [chrony-dev] SW/HW timestamping on Linux

The compensation values appear to be functioning as expected. Thanks, Denny > On Jan 10, 2017, at 03:21, Miroslav Lichvar wrote: > > Anyway, could you please confirm the new txcomp and rxcomp options in > the hwtimestamp directive work for you as expected?

Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 10, 2017, at 03:21, Miroslav Lichvar wrote: > > On Mon, Jan 09, 2017 at 09:02:14AM -0800, Denny Page wrote: >> The program up if you want to have a look. I am still working on the >> guidelines for use. >> >> https://github.com/dennypage/ethtscal >> >> My testing has been principall

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Jan 09, 2017 at 09:02:14AM -0800, Denny Page wrote: > The program up if you want to have a look. I am still working on the > guidelines for use. > > https://github.com/dennypage/ethtscal > > My testing has been principally focused on the i354. I will also be testing > the i211, which

Re: [chrony-dev] SW/HW timestamping on Linux

The program up if you want to have a look. I am still working on the guidelines for use. https://github.com/dennypage/ethtscal My testing has been principally focused on the i354. I will also be testing the i211, which I believe should be very close to the i210, but I need to finish the i354

Re: [chrony-dev] SW/HW timestamping on Linux

I have a program that will test this for you. Give me a few days to finish cleaning it up. Denny > On Jan 06, 2017, at 05:04, Miroslav Lichvar wrote: > > Could be the measured peer delay used to check if the combined TX/RX > compensation is correct? If I have two identical NICs connected > di

Re: [chrony-dev] SW/HW timestamping on Linux

> On Jan 06, 2017, at 05:04, Miroslav Lichvar wrote: > > On Thu, Jan 05, 2017 at 10:15:08AM -0800, Denny Page wrote: >> Yes, comp is an abbreviation for compensation. One will always an add and >> the other is always a subtract. This is because the timestamps are defined >> as being at the php

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Jan 05, 2017 at 10:15:08AM -0800, Denny Page wrote: > Yes, comp is an abbreviation for compensation. One will always an add and the > other is always a subtract. This is because the timestamps are defined as > being at the php layer (last bit of the SFD), but the timestamps are being > t

Re: [chrony-dev] SW/HW timestamping on Linux

Yes, comp is an abbreviation for compensation. One will always an add and the other is always a subtract. This is because the timestamps are defined as being at the php layer (last bit of the SFD), but the timestamps are being taken at the mii layer. For inbound, the phy layer precedes the mii l

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Jan 02, 2017 at 10:54:16PM -0800, Denny Page wrote: > That would be great. For option names, I would suggest something like: > > hwtimestamp igb0 txcomp 178 rxcomp 448 > > txcomp is the number of nanoseconds to add to tx timestamps > rxcomp is the number of nanoseconds to subtract from

Re: [chrony-dev] SW/HW timestamping on Linux

That would be great. For option names, I would suggest something like: hwtimestamp igb0 txcomp 178 rxcomp 448 txcomp is the number of nanoseconds to add to tx timestamps rxcomp is the number of nanoseconds to subtract from rx timestamps Thanks, Denny > On Jan 02, 2017, at 05:47, Miroslav Lic

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Dec 19, 2016 at 10:36:15PM -0800, Denny Page wrote: > Yes, I think offset covers the delayAsymmetry feature. The other two, latency > for tx and rx would be very, very helpful. Any chance of that happening prior > to release? Maybe. I'll see how invasive that change would be. I have now

Re: [chrony-dev] SW/HW timestamping on Linux

Yes, I think offset covers the delayAsymmetry feature. The other two, latency for tx and rx would be very, very helpful. Any chance of that happening prior to release? Thanks, Denny > On Dec 15, 2016, at 00:30, Miroslav Lichvar wrote: > > On Wed, Dec 14, 2016 at 10:34:19PM -0800, Denny Page

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Dec 14, 2016 at 10:34:19PM -0800, Denny Page wrote: > Btw, it might be good for Chrony to support configuration for corrections > similar to what is discussed in that thread: > > delayAsymmetry - corrects for any asymmetry between the Rx and Tx paths > egressLatency - corrects the tr

Re: [chrony-dev] SW/HW timestamping on Linux

Btw, it might be good for Chrony to support configuration for corrections similar to what is discussed in that thread: delayAsymmetry - corrects for any asymmetry between the Rx and Tx paths egressLatency - corrects the transmit latency at the MAC/PHY ingressLatency - corrects the receive

Re: [chrony-dev] SW/HW timestamping on Linux

Interesting! I would not have thought of this. Both the i211 and i354 have the SDP pins. Not exactly a pre-made PPS input, but possible with some soldering and some code. Denny > On Dec 13, 2016, at 23:21, Miroslav Lichvar wrote: > > On Tue, Dec 13, 2016 at 05:03:15PM -0800, Denny Page wrote

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Dec 13, 2016 at 06:44:38PM -0800, Denny Page wrote: > Also, would it be possible to allow hardware timestamps to be used on one > interface, and software timestamps on another (without falling back to daemon > timestamps)? I tried that, but there seem to be some limitations in the kernel

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Dec 13, 2016 at 05:03:15PM -0800, Denny Page wrote: > Unfortunately, I’ve not yet received any response on the Intel list. While > it’s quite clear that both the i211 and i354 require some compensation, I > don’t have a good way to determine it other than by shots in the dark. The > i211

Re: [chrony-dev] SW/HW timestamping on Linux

Also, would it be possible to allow hardware timestamps to be used on one interface, and software timestamps on another (without falling back to daemon timestamps)? Thanks, Denny -- To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" in the subject. For help email

Re: [chrony-dev] SW/HW timestamping on Linux

Unfortunately, I’ve not yet received any response on the Intel list. While it’s quite clear that both the i211 and i354 require some compensation, I don’t have a good way to determine it other than by shots in the dark. The i211 seems to correspond with the i210 correction parameters, so I am cu

Re: [chrony-dev] SW/HW timestamping on Linux

Thank you > On Apr 11, 2017, at 12:14, Miroslav Lichvar wrote: > > On Wed, Dec 07, 2016 at 11:26:19PM -0800, Denny Page wrote: >> Are you running linuxptp to manage the clock or letting it free-run? I have >> some ptp servers, but have been focused on testing ntp with chrony managing >> the c

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Dec 07, 2016 at 11:26:19PM -0800, Denny Page wrote: > Are you running linuxptp to manage the clock or letting it free-run? I have > some ptp servers, but have been focused on testing ntp with chrony managing > the clock. The PHC is free running. As long as the system clock can stay close

Re: [chrony-dev] SW/HW timestamping on Linux

Are you running linuxptp to manage the clock or letting it free-run? I have some ptp servers, but have been focused on testing ntp with chrony managing the clock. Thanks, Denny > On Apr 11, 2017, at 11:56, Miroslav Lichvar wrote: > > I've synchronised the system clock to the PHC with > "phc2

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Dec 07, 2016 at 04:31:38PM -0800, Denny Page wrote: > > > On Dec 07, 2016, at 09:20, Miroslav Lichvar wrote: > > > > Yes, I think that might be helpful. I spent some time today comparing > > your method with the current code and at least on my system with i210 > > I see a shift in the di

Re: [chrony-dev] SW/HW timestamping on Linux

On the offset issue… one of the benefits of having so many ports, and several of hardware based NTP units (thanks Leo/Anthony!) is that I have been able to replicate the issue by connecting serval ports directly from host to host with and without an intervening switch. I’ve also been able to tes

Re: [chrony-dev] SW/HW timestamping on Linux

> On Dec 07, 2016, at 09:20, Miroslav Lichvar wrote: > > Yes, I think that might be helpful. I spent some time today comparing > your method with the current code and at least on my system with i210 > I see a shift in the distribution of the offset to one side when the > network is (heavily) loa

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Dec 06, 2016 at 11:17:26AM -0800, Denny Page wrote: > Yes, the general server that I refer to, and which the graphs are from, is > fairly active on the primary nic. It’s the network log server, does some DB, > etc. > > On the quiet system, the change has much less impact. But it was the

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Dec 06, 2016 at 10:00:40PM -0800, Denny Page wrote: > Not sure what you meant here. The current code bases all its calculations of > intervals base on the begin time of slot zero. Even if slot zero is bad, it’s > total window is used in each time interval calculation. In the new code, a

Re: [chrony-dev] SW/HW timestamping on Linux

Not sure what you meant here. The current code bases all its calculations of intervals base on the begin time of slot zero. Even if slot zero is bad, it’s total window is used in each time interval calculation. In the new code, a bad slot is never used in any way even if it is slot zero. Denny

Re: [chrony-dev] SW/HW timestamping on Linux

Yes, the general server that I refer to, and which the graphs are from, is fairly active on the primary nic. It’s the network log server, does some DB, etc. On the quiet system, the change has much less impact. But it was the general server that I was frequently seeing small spikes on. I can c

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Dec 05, 2016 at 02:35:22PM -0800, Denny Page wrote: > > > On Dec 05, 2016, at 01:05, Miroslav Lichvar wrote: > > > > If I understand your change correctly, it increases the minimum > > acceptable delay, which increases the number of combined readings and > > makes the offset more stable.

Re: [chrony-dev] SW/HW timestamping on Linux

And a visual representation. This is from the loaded system.

Re: [chrony-dev] SW/HW timestamping on Linux

Here is an example of where I am at after the change: 210 Number of sources = 6 MS Name/IP address Stratum Poll Reach LastRx Last sample === ^* 192.168.230.240 1 0 377 0+24n

Re: [chrony-dev] SW/HW timestamping on Linux

> On Dec 05, 2016, at 01:05, Miroslav Lichvar wrote: > > If I understand your change correctly, it increases the minimum > acceptable delay, which increases the number of combined readings and > makes the offset more stable. I'm worried on a busy system bad > readings could be included and disru

Re: [chrony-dev] SW/HW timestamping on Linux

On Sun, Dec 04, 2016 at 10:56:57PM -0800, Denny Page wrote: > I’m still chasing the offset issue. However on the spikes/noise issue, the > patch below addresses the noise and low level spikes that I was seeing. I was > seeing errors of 2-60ns off in the average offset for the nic clock, which >

Re: [chrony-dev] SW/HW timestamping on Linux

I’m still chasing the offset issue. However on the spikes/noise issue, the patch below addresses the noise and low level spikes that I was seeing. I was seeing errors of 2-60ns off in the average offset for the nic clock, which this corrects. I’m still seeing the occasional larger spike, but thi

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 28, 2016 at 09:22:42AM -0800, Denny Page wrote: > Before I try and make a case to the driver and hardware folk, I think I need > to be able to explain how stamps on both two linux systems can sometimes be > in agreement with stamps on the second interface and sometimes not. Given > j

Re: [chrony-dev] SW/HW timestamping on Linux

> On Nov 28, 2016, at 01:01, Miroslav Lichvar wrote: > > If you are sure the error doesn't come from the switch, I suspect it's > a HW or driver issue. It seems the drivers need to have some > timestamping-specific magic. Look at the following commit for > instance. > > http://git.kernel.org/cgi

Re: [chrony-dev] SW/HW timestamping on Linux

On Fri, Nov 25, 2016 at 05:11:08PM -0800, Denny Page wrote: > I believe the stddev issue with multiple interfaces may have been leftover > sampling from prior runs. I cleared all the point data, and things looked > better. Sorry about that. > > The priority issue remains, but I don’t think it’s

Re: [chrony-dev] SW/HW timestamping on Linux

On Sun, Nov 27, 2016 at 01:06:20PM -0800, Denny Page wrote: > Here are the various test restuls: > > igb0 @ 1Gb; igb3 @ 100Mb direct connect: 192.168.230.245 shows offset of > +1230ns > igb0 @ 100Mb; igb3 @ 100Mb, direct connect: 192.168.230.245 shows no offset > igb0 @ 1Gb, igb3 @ 1Gb via

Re: [chrony-dev] SW/HW timestamping on Linux

I still have one significant offset mystery with the latest version. In the tests described below, the following IP addresses are involved: 192.168.230.2 Linux box. igb0 connected to main switch at 1Gb. 192.168.230.3 Linux box. igb0 connected to main switch, speed as described in tests. igb3

Re: [chrony-dev] SW/HW timestamping on Linux

I believe the stddev issue with multiple interfaces may have been leftover sampling from prior runs. I cleared all the point data, and things looked better. Sorry about that. The priority issue remains, but I don’t think it’s a big deal per se. When running with scheduling priority (-P 9), I se

Re: [chrony-dev] SW/HW timestamping on Linux

That cured my issue with offset between the hosts. > On Nov 24, 2016, at 06:57, Miroslav Lichvar wrote: > > On Thu, Nov 24, 2016 at 06:46:36AM -0800, Denny Page wrote: >> I do not have xleave enabled. Can you explain more as to why you would >> expect to see such an offset without xleave? > >

Re: [chrony-dev] SW/HW timestamping on Linux

This makes sense. Thank you. I will enable xleave and retest shortly. Denny > On Nov 24, 2016, at 06:57, Miroslav Lichvar wrote: > > On Thu, Nov 24, 2016 at 06:46:36AM -0800, Denny Page wrote: >> I do not have xleave enabled. Can you explain more as to why you would >> expect to see such an o

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 24, 2016 at 06:46:36AM -0800, Denny Page wrote: > I do not have xleave enabled. Can you explain more as to why you would expect > to see such an offset without xleave? The offset is calculated from four timestamps. Two local and two remote. Without xleave the peer can't deliver the TX

Re: [chrony-dev] SW/HW timestamping on Linux

I do not have xleave enabled. Can you explain more as to why you would expect to see such an offset without xleave? Thanks, Denny > On Nov 24, 2016, at 05:11, Miroslav Lichvar wrote: > > That suggests the interleaved mode is not working. Are the peers specified > with the xleave option? -- T

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Nov 23, 2016 at 03:24:56PM -0800, Denny Page wrote: > I am now seeing better standard deviations with hardware timestamping than > software timestamping. Thank you. > > Couple of caveats: > - I need to disable priority scheduling (-P). With priority scheduling, > software stamps still

Re: [chrony-dev] SW/HW timestamping on Linux

I am now seeing better standard deviations with hardware timestamping than software timestamping. Thank you. Couple of caveats: - I need to disable priority scheduling (-P). With priority scheduling, software stamps still have lower stddev. - I can only use a single ethernet interface. With

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Nov 23, 2016 at 08:09:15AM -0800, Denny Page wrote: > Miroslav, what version of the kernel are you using for testing? And what are > you using for scheduling priority? I'm now on 4.8.3 from Fedora 25 and default priority, whatever that is. -- Miroslav Lichvar -- To unsubscribe email c

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, what version of the kernel are you using for testing? And what are you using for scheduling priority? Denny -- To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" in the subject. For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the subj

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 22, 2016 at 09:06:08AM -0800, Denny Page wrote: > I tested with that earlier. Almost all the timestamps were rejected. But I > was using poll 0. I’ll have to test again with a longer polling interval. Rejected meaning measurements with 'D H' had a failing test bit in the log? The pol

Re: [chrony-dev] SW/HW timestamping on Linux

On Fri, Nov 18, 2016 at 03:37:07PM +0100, Miroslav Lichvar wrote: > On Thu, Nov 17, 2016 at 05:44:23PM -0800, Denny Page wrote: > > Although reduced, I’m still seeing spikes with the patch below. > > I'm not sure what could be wrong at this point. Maybe it really is a > kernel or HW issue. I'm won

Re: [chrony-dev] SW/HW timestamping on Linux

I tested with that earlier. Almost all the timestamps were rejected. But I was using poll 0. I’ll have to test again with a longer polling interval. Denny > On Nov 22, 2016, at 07:43, Miroslav Lichvar wrote: > > Can you confirm that with the patch I sent you earlier > which waits with poll()?

Re: [chrony-dev] SW/HW timestamping on Linux

Perfect. Thank you. Denny > On Nov 22, 2016, at 07:34, Miroslav Lichvar wrote: > > I've pushed to the git an implementation of the idea I proposed > earlier, which assumes symmetric position of UDP data in received and > transmitted packets. You can modify the calculation or hardcode the RX >

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 08:53:51PM -0800, Denny Page wrote: > I would not have expected this, and I haven’t dug into it, but looking at the > time to generate timestamps for pcap, I am seeing times of more than a > second(!). I am wondering what happens if we get to the point of sending the > ne

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 09:29:13PM -0800, Denny Page wrote: > Just to make sure I test correctly, would you mind creating a diff for a test > version for me? One that a can plug a hard coded number of nanoseconds > correction for just packet receive? I've pushed to the git an implementation of t

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, Just to make sure I test correctly, would you mind creating a diff for a test version for me? One that a can plug a hard coded number of nanoseconds correction for just packet receive? Thanks, Denny > On Nov 21, 2016, at 17:26, Denny Page wrote: > > I may hard code a correction f

Re: [chrony-dev] SW/HW timestamping on Linux

I would not have expected this, and I haven’t dug into it, but looking at the time to generate timestamps for pcap, I am seeing times of more than a second(!). I am wondering what happens if we get to the point of sending the next request before we have received the transmit timestamp for the pr

Re: [chrony-dev] SW/HW timestamping on Linux

I checked this. It appears to be completely symmetrical at the switch level. For both the 1Gb and 100Mb ports, the delta I see for NTP packets is a very consistent 912ns on a 1Gb mirror port. This is exactly as expected: preamble: 56 bits SFD: 8 bits MAC dest: 48 bits MAC src: 48 bits

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 11:25:10AM -0800, Denny Page wrote: > On Nov 21, 2016, at 10:59, Miroslav Lichvar wrote: > > So, in order to make this as simple as possible, we could make an > > assumption that received packets have the same format as transmitted > > packets. At least with VLANs this migh

Re: [chrony-dev] SW/HW timestamping on Linux

> On Nov 21, 2016, at 10:59, Miroslav Lichvar wrote: > > On Mon, Nov 21, 2016 at 10:28:26AM -0800, Denny Page wrote: >> The only layer we know the size of is the UDP layer. The Ethernet layer can >> have VLAN tag. Yes, it’s only 32 bits, but error is error. The IP layer may >> have option head

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 10:28:26AM -0800, Denny Page wrote: > The only layer we know the size of is the UDP layer. The Ethernet layer can > have VLAN tag. Yes, it’s only 32 bits, but error is error. The IP layer may > have option headers. This is rare with IPv4, but not with IPv6. With TX timest

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, It was 300ns. I’ve tried several different switches, each with differing results, although I didn’t run long enough to be be absolutely sure they would not settle to the same number. With my main Cisco switch I’m seeing the 8 hour average at 298.858ns. Such a precise number. I don’t h

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, We actually don’t know the length of the data at layer 2. We can’t even guarantee the length at layer 3. Dispensing with layer numbers from now on. We have three levels we have to concern ourselves with: Ethernet IP UDP The only layer we know the size of is the UDP layer. The E

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, This is called a cut-through, and is not the default. The reason it isn’t the default is because it propagates invalid frames (runts, run-ons, crc errors, etc.) beyond the source port. Store and forward is the default. Cut-though generally doesn’t work with port speed mismatch and I’v

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 21, 2016 at 09:38:36AM -0800, Denny Page wrote: > Given host A on a 1Gb link and host B on a 100Mb link: > > A -> Switch transmission time 754ns > Switch -> B transmission time 7540ns > Total time 8294ns As I understand it, modern switches don't wait until they have whole packet

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, There is no correction necessary. The speed change itself does not have any impact because the transmission time is symmetrical. Given host A on a 1Gb link and host B on a 100Mb link: A -> Switch transmission time 754ns Switch -> B transmission time 7540ns Total time 8294ns B

Re: [chrony-dev] SW/HW timestamping on Linux

On Sun, Nov 20, 2016 at 05:18:56PM -0800, Denny Page wrote: > From that article: > > • A preamble timestamp is struck as near to the start of the packet as > possible. The preferred point follows the last bit of the preamble and > start-of-frame (STF) octet and before the first octet of th

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I found the problem. The hardware receive timestamps are incorrect. It doesn’t matter if there is a switch involved or not. NTP depends upon round trip latency being symmetrical for it’s accuracy, both in local network and in remote networks. To help ensure this, NTP uses preamble ti

Re: [chrony-dev] SW/HW timestamping on Linux

I’ll have to come back to this after the offset issue is resolved. Denny > On Nov 18, 2016, at 06:37, Miroslav Lichvar wrote: > > On Thu, Nov 17, 2016 at 05:44:23PM -0800, Denny Page wrote: >> Although reduced, I’m still seeing spikes with the patch below. > > I'm not sure what could be wron

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I believe that the hardware NTP device, chrony, or both, are striking/calculating timestamps incorrectly. I have a test in mind that will allow me to determine if this is correct, and if so which. Back to you soon. Denny > On Nov 18, 2016, at 00:00, Miroslav Lichvar wrote: > > On

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 17, 2016 at 05:44:23PM -0800, Denny Page wrote: > Although reduced, I’m still seeing spikes with the patch below. I'm not sure what could be wrong at this point. Maybe it really is a kernel or HW issue. I'm wondering what would be the best way to confirm or reject that idea. -- Miros

Re: [chrony-dev] SW/HW timestamping on Linux

On Thu, Nov 17, 2016 at 05:49:44PM -0800, Denny Page wrote: > This port speed differential appears to result in a asymmetry in > transmit/receive time which significantly affects the calculations. If I lock > the monitor host port at 100Mb, all three units show precise synchronization, > both wi

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I’ve found the issue with the 2.2us offset between a locally attached vs across the switch. It’s not the switch. The act of crossing the switch itself seems to be negligible. The problem appears to be one of asymmetry arising from port speed mismatch. The hardware NTP device uses 10/

Re: [chrony-dev] SW/HW timestamping on Linux

Although reduced, I’m still seeing spikes with the patch below. Denny > On Nov 16, 2016, at 00:53, Miroslav Lichvar wrote: > > On Tue, Nov 15, 2016 at 08:24:37PM -0800, Denny Page wrote: >> With the latest drop in the repo, I’m still seeing the wild spikes in the >> standard deviation with h

Re: [chrony-dev] SW/HW timestamping on Linux

Yep. Seeing lots of that. ~70% of the samples for the locally attached unit. Denny > On Nov 16, 2016, at 23:39, Miroslav Lichvar wrote: > > On Wed, Nov 16, 2016 at 11:24:14PM -0800, Denny Page wrote: >> To be clear, would I still expect to see ‘D H’ in the measurements log with >> this change

Re: [chrony-dev] SW/HW timestamping on Linux

On Wed, Nov 16, 2016 at 11:24:14PM -0800, Denny Page wrote: > To be clear, would I still expect to see ‘D H’ in the measurements log with > this change? Yes, but the second bit in the column with four test bits should be always zero. > Thanks, > Denny > > > > On Nov 16, 2016, at 00:53, Mirosla

Re: [chrony-dev] SW/HW timestamping on Linux

To be clear, would I still expect to see ‘D H’ in the measurements log with this change? Thanks, Denny > On Nov 16, 2016, at 00:53, Miroslav Lichvar wrote: > > Hm, the fix helped with the spikes I was seeking. Did we rule out the > possibility that in your case the spikes are due to the other

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, I’m sorry, I didn’t mean to imply that the spikes were an unrelated to the ordering issue. I do believe that the spikes are related to the hardware timestamp ordering issue. I’ll try to set up some of the tests you’ve requested later this evening, but it may take a day for me to have

Re: [chrony-dev] SW/HW timestamping on Linux

Yes, all ports on the monitoring system are identical. The i354 is a 4 port chip, and all the ethernet ports on the monitoring unit are connected through that same chip. The general server has both I354 and I211 chips. I shut everything down on the server to conduct a test. It didn’t matter whi

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 15, 2016 at 09:39:40PM -0800, Denny Page wrote: > > While I can get my head around a differential of 250ns resulting from the > switch, I’m finding it very difficult to believe the almost 2500ns > differential that appears when hardware time stamping is enabled. > > Any thoughts on

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 15, 2016 at 08:24:37PM -0800, Denny Page wrote: > With the latest drop in the repo, I’m still seeing the wild spikes in the > standard deviation with hardware time stamping against the fast responding > hardare units. I'm also still seeing a better base deviation using software > tim

Re: [chrony-dev] SW/HW timestamping on Linux

On Tue, Nov 15, 2016 at 08:14:28PM -0800, Denny Page wrote: > The chip is actually a I354, which is slightly different than the I350, but I > don’t think it matters much. I also have interfaces with I211 chips, and the > ordering issue appears to happen there as well. I don’t think the sleep afte

Re: [chrony-dev] SW/HW timestamping on Linux

Miroslav, There is an additional issue, which is perhaps unrelated to the other issues discussed. Background: Each target IP is a hardware based NTP server. IPs 240 and 244 are across the switch. Chrony is synchronizing with these two IPs. Incremental latency from the switch is approximately ~

Re: [chrony-dev] SW/HW timestamping on Linux

With the latest drop in the repo, I’m still seeing the wild spikes in the standard deviation with hardware time stamping against the fast responding hardare units. I'm also still seeing a better base deviation using software timestamps against them as well. I do see better results with hardware

Re: [chrony-dev] SW/HW timestamping on Linux

The chip is actually a I354, which is slightly different than the I350, but I don’t think it matters much. I also have interfaces with I211 chips, and the ordering issue appears to happen there as well. I don’t think the sleep after the send is going to affect the order of the timestamp and resp

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 14, 2016 at 03:40:32PM +0100, Miroslav Lichvar wrote: > On Sat, Nov 12, 2016 at 10:36:55AM -0800, Denny Page wrote: > > Here is a reasonable visual representation of what I am seeing. The section > > on the left (before 8:00) is with hardware timestamping, while the section > > on the

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 14, 2016 at 08:59:17PM -0800, Denny Page wrote: > I tested with a usleep(100) following the sendmsg() call. This didn’t appear > to have any impact. Was the usleep() intended to influence the order of > timestamp vs. server response messages? Yes, that was the idea. Could you try inc

Re: [chrony-dev] SW/HW timestamping on Linux

I tested with a usleep(100) following the sendmsg() call. This didn’t appear to have any impact. Was the usleep() intended to influence the order of timestamp vs. server response messages? Denny > On Nov 14, 2016, at 11:04, Miroslav Lichvar wrote: > > usleep() should be called after sendmsg(

Re: [chrony-dev] SW/HW timestamping on Linux

Setting noselect for all sources seems to have no impact on standard deviation. Disabling all forms of dynamic tic (CONFIG_HZ_PERIODIC=y) seems to have some effect in reducing the number of “D H”, but does not appear to have much of an impact on the standard deviation. I am returning CONFIG_NO_H

Re: [chrony-dev] SW/HW timestamping on Linux

On Mon, Nov 14, 2016 at 10:47:52AM -0800, Denny Page wrote: > I’m not sure I understand what effect a delay in NIO_Linux_RequestTxTimestamp > would have. That I see, NIO_Linux_RequestTxTimestamp builds a control message > structure but does not make any level 2 calls. Introducing a delay here >

Re: [chrony-dev] SW/HW timestamping on Linux

I’m not sure I understand what effect a delay in NIO_Linux_RequestTxTimestamp would have. That I see, NIO_Linux_RequestTxTimestamp builds a control message structure but does not make any level 2 calls. Introducing a delay here should be the same as introducing a delay one level up in NIO_SendPa

  1   2   >