Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Denny Page 
Miroslav,

This is called a cut-through, and is not the default. The reason it isn’t the 
default is because it propagates invalid frames (runts, run-ons, crc errors, 
etc.) beyond the source port. Store and forward is the default. Cut-though 
generally doesn’t work with port speed mismatch and I’ve never seen it used 
with 100Mb. This really isn’t something you have to worry about. 

FWIW, assuming two 1Gb ports on the same switch, the one-way transmission time 
with cut-through would go from 1504ns to 800ns. However the round trip would 
still be symmetrical.

Denny


> On Nov 21, 2016, at 09:52, Miroslav Lichvar  wrote:
> 
> As I understand it, modern switches don't wait until they have whole
> packet before forwading it to another port. They just need to wait for
> the part that contains the destination MAC address. So the total time
> should be closer to 7540 ns.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Miroslav Lichvar 
On Mon, Nov 21, 2016 at 10:28:26AM -0800, Denny Page wrote:
> The only layer we know the size of is the UDP layer. The Ethernet layer can 
> have VLAN tag. Yes, it’s only 32 bits, but error is error. The IP layer may 
> have option headers. This is rare with IPv4, but not with IPv6.

With TX timestamping the kernel gives us ethernet frames and we have to
painfully extract the UDP data from them. IPv6 Options are not
currently supported.

So, in order to make this as simple as possible, we could make an
assumption that received packets have the same format as transmitted
packets. At least with VLANs this might work more often than not.

> The only way to correct the hardware timestamp is to know the entire length 
> of the frame at the Ethernet level. I don’t see how this happens without 
> using raw sockets.

I'd rather avoid raw sockets, at least for now. Even if the length
will be off by couple bytes, it should still be more accurate than SW
timestamps.

> The formula would be: timestamp += (ether_bytes + 4) * 8 / link_bps.

Ok, so if we store offset of UDP data from the beginning of a
transmitted frame at layer 2, it would be this?

(udp_data_offset + NTP packet length + 4)

-- 
Miroslav Lichvar

-- 
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Denny Page 
Miroslav,

There is no correction necessary. The speed change itself does not have any 
impact because the transmission time is symmetrical.

Given host A on a 1Gb link and host B on a 100Mb link:

  A -> Switch transmission time 754ns
  Switch -> B transmission time 7540ns
  Total time 8294ns

  B -> Switch transmission time 7540ns
  Switch -> transmission time 754ns
  Total time 8294ns

The propagation delay is symmetrical.

Unless there are a backlog of packets on one of the involved ports, the switch 
has zero impact on accuracy.

Denny



> On Nov 21, 2016, at 00:30, Miroslav Lichvar  wrote:
> 
> In your case you will still need to have some correction for the extra
> delay due the switch in the 100mbit->1gbit direction and I suspect
> switches in general will be the biggest trouble when trying to get the
> best accuracy.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Denny Page 
Miroslav,

We actually don’t know the length of the data at layer 2. We can’t even 
guarantee the length at layer 3. Dispensing with layer numbers from now on.

We have three levels we have to concern ourselves with:

  Ethernet
  IP
  UDP

The only layer we know the size of is the UDP layer. The Ethernet layer can 
have VLAN tag. Yes, it’s only 32 bits, but error is error. The IP layer may 
have option headers. This is rare with IPv4, but not with IPv6.

The only way to correct the hardware timestamp is to know the entire length of 
the frame at the Ethernet level. I don’t see how this happens without using raw 
sockets.

The formula would be: timestamp += (ether_bytes + 4) * 8 / link_bps.

Denny


> On Nov 21, 2016, at 00:30, Miroslav Lichvar  wrote:
> 
> We know the length of the
> transmitted data at layer 2. Maybe we could use the same length of the
> headers for received packets on the same interface? The link speed
> should not be too difficult to get. Can you suggest a formula?

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Miroslav Lichvar 
On Mon, Nov 21, 2016 at 09:38:36AM -0800, Denny Page wrote:
> Given host A on a 1Gb link and host B on a 100Mb link:
> 
>   A -> Switch transmission time 754ns
>   Switch -> B transmission time 7540ns
>   Total time 8294ns

As I understand it, modern switches don't wait until they have whole
packet before forwading it to another port. They just need to wait for
the part that contains the destination MAC address. So the total time
should be closer to 7540 ns.

>   B -> Switch transmission time 7540ns
>   Switch -> transmission time 754ns
>   Total time 8294ns

Yes, in this case the switch has to wait as it can't connect a 100mbit
input directly to a 1gbit output.

> The propagation delay is symmetrical.
> 
> Unless there are a backlog of packets on one of the involved ports, the 
> switch has zero impact on accuracy.

I think your own experiments showed there is a 350 ns offset :). That
would be 700 extra nanoseconds in the B->A direction.

-- 
Miroslav Lichvar

-- 
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Denny Page 

> On Nov 21, 2016, at 10:59, Miroslav Lichvar  wrote:
> 
> On Mon, Nov 21, 2016 at 10:28:26AM -0800, Denny Page wrote:
>> The only layer we know the size of is the UDP layer. The Ethernet layer can 
>> have VLAN tag. Yes, it’s only 32 bits, but error is error. The IP layer may 
>> have option headers. This is rare with IPv4, but not with IPv6.
> 
> With TX timestamping the kernel gives us ethernet frames and we have to
> painfully extract the UDP data from them.

I didn’t know this. Thanks. If we already have the raw Ethernet frame in hand, 
then raw sockets are not necessary. I’ll have to look at the code.


> So, in order to make this as simple as possible, we could make an
> assumption that received packets have the same format as transmitted
> packets. At least with VLANs this might work more often than not.

It really doesn’t have anything to do with the packet as we sent it. VLANS and 
IP options can be added along the way. We only need to look at the packet as 
received in order to correct the receive timestamp. The transmit timestamp is 
already correct.


>> The formula would be: timestamp += (ether_bytes + 4) * 8 / link_bps.
> 
> Ok, so if we store offset of UDP data from the beginning of a
> transmitted frame at layer 2, it would be this?
> 
>   (udp_data_offset + NTP packet length + 4)

For the correction, we don’t care where the UDP data is. All we want is the 
length of the Ethernet frame. If this isn’t handed to us when we are given the 
raw frame, we can pull the length out of the frame. I’ll have to look at the 
code.

Denny


--
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Miroslav Lichvar 
On Mon, Nov 21, 2016 at 11:25:10AM -0800, Denny Page wrote:
> On Nov 21, 2016, at 10:59, Miroslav Lichvar  wrote:
> > So, in order to make this as simple as possible, we could make an
> > assumption that received packets have the same format as transmitted
> > packets. At least with VLANs this might work more often than not.
> 
> It really doesn’t have anything to do with the packet as we sent it. VLANS 
> and IP options can be added along the way. We only need to look at the packet 
> as received in order to correct the receive timestamp. The transmit timestamp 
> is already correct.

The trouble is we have only transmitted ethernet frames, not received.

-- 
Miroslav Lichvar

-- 
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Denny Page 
I checked this. It appears to be completely symmetrical at the switch level. 
For both the 1Gb and 100Mb ports, the delta I see for NTP packets is a very 
consistent 912ns on a 1Gb mirror port. This is exactly as expected:

  preamble: 56 bits
  SFD: 8 bits
  MAC dest: 48 bits
  MAC src: 48 bits
  Length: 16 bits
  IP/UDP/NTP: 608 bits
  FCS: 32 bits
  IPG: 96 bits

Total of 912 bits, which equates to 912ns on a 1Gb connection.

NB: The packet cannot be forwarded until it is received, so the preamble and 
IPG count when looking at packet to packet deltas when mirroring.

I’ll have to look at the driver/kernel next. It may just be an inherent offset 
due to software timestamping. I may hard code a correction for the hardware 
timestamps in chrony for testing.

Denny


> On Nov 21, 2016, at 10:50, Denny Page  wrote:
> 
> I am going to set up a span port to capture the packets on both interfaces 
> which may give an indication.


--
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Denny Page 
I would not have expected this, and I haven’t dug into it, but looking at the 
time to generate timestamps for pcap, I am seeing times of more than a 
second(!). I am wondering what happens if we get to the point of sending the 
next request before we have received the transmit timestamp for the prior one. 
Given the speed of the units I’m testing with, it appears to be possible to 
have received the second response before the first transmit timestamp.

Have you tested against a fast server with "minpoll 0 max poll 0”? 

Just a thought.

Denny


> On Nov 18, 2016, at 06:37, Miroslav Lichvar  wrote:
> 
> On Thu, Nov 17, 2016 at 05:44:23PM -0800, Denny Page wrote:
>> Although reduced, I’m still seeing spikes with the patch below.
> 
> I'm not sure what could be wrong at this point. Maybe it really is a
> kernel or HW issue. I'm wondering what would be the best way to
> confirm or reject that idea.


--
To unsubscribe email chrony-dev-requ...@chrony.tuxfamily.org with "unsubscribe" 
in the subject.
For help email chrony-dev-requ...@chrony.tuxfamily.org with "help" in the 
subject.
Trouble?  Email listmas...@chrony.tuxfamily.org.

Re: [chrony-dev] SW/HW timestamping on Linux

2016-11-21 Thread Denny Page 
Miroslav,

Just to make sure I test correctly, would you mind creating a diff for a test 
version for me? One that a can plug a hard coded number of nanoseconds 
correction for just packet receive?

Thanks,
Denny


> On Nov 21, 2016, at 17:26, Denny Page  wrote:
> 
>  I may hard code a correction for the hardware timestamps in chrony for 
> testing.

[chrony-dev] chrony-2.4.1 released

2016-11-21 Thread Miroslav Lichvar 
chrony-2.4.1 is now available. It's a bugfix release, which fixes a
crash with the smoothtime directive when used with the leaponly option
to perform a synchronised leap smear, processing of kernel timestamps
on non-Linux systems, and few other bugs.

The sources can be downloaded here:
http://download.tuxfamily.org/chrony/chrony-2.4.1.tar.gz

MD5 and SHA1 sums:
d08dd5a7d79a89891d119adcccb4397d  chrony-2.4.1.tar.gz
b412375ca90dbef653ad00534f8b73b825e396d4  chrony-2.4.1.tar.gz

-- 
Miroslav Lichvar


signature.asc
Description: PGP signature