> From: Terje Mathisen <[EMAIL PROTECTED]> > Date: Sun, 07 Dec 2008 11:44:18 +0100 > Sender: [EMAIL PROTECTED] > > > Maarten Wiltink wrote: > > "Unruh" <[EMAIL PROTECTED]> wrote in message > > news:[EMAIL PROTECTED] > > [...] > >>> What should I be doing to get 20 us? Buy all new computers with > >>> gigabit Ethernet? > >> I suspect buying better switches. And it looks to me like you should > >> definitely NOT go to gigabit Ethernet if you want good timing. > > > > Seeing as how gigabit Ethernet uses jumbo frames to reach decent > > throughput, I suspect it's subject to higher jitter at the network > > level. NTP frames are short, but if it's contending with a large > > download, there will be a relatively large unpredictable delay on > > the line. > > I wrote a very similar msg, then I started to consider the math: > > 100 Mbit uses 1.5 kB packets, while Gbit jumbo frames are typically 9 > kB, right? > > The thing is that as long as the jumbo frames are less than 10 times the > size of regular packets, the wire will be busy for a shorter time than > for a 1.5 K packet on 100 Mbit! > > Hybrid interrupt/polled hardware & drivers are a much more likely > culprit: If the hardware waits a little bit before interrupting, hoping > to be able to bunch up several packets in a single task, then jitter > will suffer badly.
Before this gets too far off track, standards compliant Ethernet uses frames of 1500 bytes, regardless of speed. This is true for 10M or 10G Ethernet. Most modern cards have support for jumbo frames. These are, by definition, non-standard, and are available only when specifically configured. Unless you turn them on, frames are 1500 data bytes plus framing of up to 22 bytes. When jumbos are enabled, the size currently recommended by the US federal Joint Engineering Team is 9000 bytes to allow the transmission of 8K of actual user data when allowing for framing, VLAN and MPLS tags and such with a comfortable margin over that. (Most modern systems page memory in 8K chunks, so this optimizes performance.) So, if you have not set your card to use jumbo frames, you are not using them. You are using 1500 byte MTUs. (On a Unix system, ifconfig(1) should show the MTU.) Interrupt coalescing is available on many modern 1G or faster Ethernet interfaces. It is probably universal in any PCI-e 10G interface. It requires driver support, but is available on most modern systems. It will delay interrupts for a defined time so that only a single interrupt is needed for multiple packets. For 10GE cards, a default of about 30-60 microseconds is normal, although this is usually controllable in around 1 microsecond units. 1G cards based on Intel chips may support coalescing in similar increments, but the default delay is highly variable. This could explain some of the jitter. But the jitter introduced should be fairly low, so I doubt it can explain what is seen here. Out network uses Intel based 1G interfaces and we do not see any excessive jitter on FreeBSD. But the interrupt delay for coalescing is driver controlled and other OSes my use a greater delay. In FreeBSD, the delay defaults to 0 and may be adjusted with a sysctl. On other OSes, you will need to read the documentation to see what, if any, delay is being used. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 _______________________________________________ questions mailing list [email protected] https://lists.ntp.org/mailman/listinfo/questions
