Thanks John. This makes sense to me. These controllers are LOM so configuration
changes are limited. I wouldn't be surprised if I am in a PCIe x1
configuration, and I'll do a bit more digging to find out more. On one device I
can plug in an external NIC so I'll source an i350 based device to see if it
makes a difference.
I am also getting some eval units in for 10GigE, but I can also can do some
GigE testing. These units are based on Sandy Bridge and I will confirm that the
x8 PCIe slot is on the north bridge. I am evaluating 82599 and X540
controllers. Hopefully I don't run into similar delays.
Here is device 1 (no external PCIe slots) ...
-------------------- cut ------------------
# lspci -t
-[0000:00]-+-00.0
+-02.0
+-02.1
+-1c.0-[0000:02]----00.0
+-1c.1-[0000:03]----00.0
+-1c.2-[0000:04]----00.0
+-1c.3-[0000:05]----00.0
+-1c.4-[0000:06]----00.0
+-1d.0
+-1d.1
+-1d.7
+-1e.0-[0000:01]--
+-1f.0
+-1f.1
+-1f.2
\-1f.3
# lspci
00:00.0 Host bridge: Intel Corporation N10 Family DMI Bridge (rev 02)
00:02.0 VGA compatible controller: Intel Corporation N10 Family Integrated
Graphics Controller (rev 02)
00:02.1 Display controller: Intel Corporation N10 Family Integrated Graphics
Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1
(rev 03)
00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2
(rev 03)
00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 3
(rev 03)
00:1c.3 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 4
(rev 03)
00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5
(rev 03)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI
Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI
Controller #2 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI
Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f3)
00:1f.0 ISA bridge: Intel Corporation 82801HEM (ICH8M) LPC Interface Controller
(rev 03)
00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE
Controller (rev 03)
00:1f.2 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) SATA IDE
Controller (rev 03)
00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 03)
02:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection
03:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection
04:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection
05:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection
06:00.0 Network controller: Intel Corporation Device 0091 (rev 34)
-------------------- cut ------------------
... and here is device 2 ....
-------------------- cut ------------------
# lspci -t
-[0000:00]-+-00.0
+-01.0-[0000:01]--+-00.0
| \-00.1
+-02.0
+-1c.0-[0000:02]----00.0
+-1c.1-[0000:03]----00.0
+-1c.2-[0000:04]----00.0
+-1c.3-[0000:05]----00.0
+-1c.4-[0000:06]----00.0
+-1c.5-[0000:07]----00.0
+-1d.0
+-1d.7
+-1e.0-[0000:08]--
+-1f.0
+-1f.1
+-1f.2
\-1f.3
# lspci
00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03)
00:01.0 PCI bridge: Intel Corporation 4 Series Chipset PCI Express Root Port
(rev 03)
00:02.0 VGA compatible controller: Intel Corporation 4 Series Chipset
Integrated Graphics Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 1 (rev
01)
00:1c.1 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 2 (rev
01)
00:1c.2 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 3 (rev
01)
00:1c.3 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 4 (rev
01)
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express
Port 5 (rev 01)
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express
Port 6 (rev 01)
00:1d.0 USB Controller: Intel Corporation N10/ICH7 Family USB UHCI Controller
#1 (rev 01)
00:1d.7 USB Controller: Intel Corporation N10/ICH 7 Family USB2 EHCI Controller
(rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface
Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller
(rev 01)
00:1f.2 IDE interface: Intel Corporation N10/ICH7 Family SATA IDE Controller
(rev 01)
00:1f.3 SMBus: Intel Corporation N10/ICH 7 Family SMBus Controller (rev 01)
01:00.0 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AT CX4
Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AT CX4
Network Connection (rev 01)
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
-------------------- cut ------------------
Tnx, Fred.
-----Original Message-----
From: Ronciak, John [mailto:john.ronc...@intel.com]
Sent: September-07-12 4:06 PM
To: Fred Klassen; e1000-devel@lists.sourceforge.net
Subject: RE: [E1000-devel] ] first 3 packets of a burst delayed
Hello Fred,
We think the difference is coming from the fact that the ones showing the delay
are PCIe x1 devices. If they are connected to slots going through the south
bridge could account for the initial delay. Once the data starts flowing the
delay goes away, which is what you are seeing. The other devices mentioned are
not x1 devices which means they are connected through a shorted path to the
root complex. You can check the connections by using 'lspci-t' which will show
how the devices are connected within the system. The x1 also means that moving
it to other slots may not help as the slots will be throttled to account for
the x1 speed difference. If you are really concerned about latency server type
devices should be used such as the i350 devices support by the igb driver.
Cheers,
John
> -----Original Message-----
> From: Fred Klassen [mailto:fklas...@appneta.com]
> Sent: Friday, September 07, 2012 1:18 PM
> To: e1000-devel@lists.sourceforge.net
> Subject: [E1000-devel] ] first 3 packets of a burst delayed
>
> Hi.
>
> We have a requirement for highly accurate measurements of small bursts
> and datagrams. On some of our appliances we don't have the following
> issue, but when attempting to use 82574 and 82583 controllers, we are
> seeing TX delays of the first 3 packets in a burst. It's quite
> dramatic when graphed against other controllers (non e1000e, e.g.
> 82598, Marvell), but the delays are quite significant. A datagram (or
> first packet in a burst) is delayed 150 usec, second packet delayed 80
> usec, third delayed 40 usec and no delay on any subsequent packets.
> It's almost like a Nagel algorithm.
>
> These delays are quite significant for our application and it would
> be great to get rid of them. I tried all module parameters and I tried
> modifying some defaults such as BURST_RDTR. Is there anything we can
> do to eliminate these delays?
>
> --
> Fred Klassen
> Founder, AppNeta
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
