Hello,
we are running into an issue with intel 82599EB 10-Gigabit, where ping
round trip times on a local network increase from 0.1ms to >100ms when
the nics are put under a moderate load. This causes all sorts of
latency related problems.
Interestingly there seems to be no packet loss, as ping reports 0%
packet loss.
When testing with iperf I'm unable to reproduce the issue; iperf measures
9.5Gbit bandwith and ping reports 0.1ms rtt.
However, the problem is reproducable with a apache / ab combination
When running apache (geared for large downloads) on the server and
ab in the same network as client it is easy to trigger the problem.
It should be noted however that the problem only exists when ab
is run with a largish concurrency (-c 50).
Our test setup is as follows:
- host1 runs a webserver (dltest.omroep.nl); serving a 100Mbyte test file
- host2 acts as the ab client. The ab command line is
ab -n 100 -c 50 http://dltest.omroep.nl/100m
- host3 acts as an observer, performing ping:
sudo ping -c 100 -i 0.001 -q dltest.omroep.nl
The ping output (when host1 is under load) is consistenly >100 ms:
$ sudo ping -c 100 -i 0.001 -q dltest.omroep.nl
--- dltest1afp.omroep.nl ping statistics ---
PING dltest1afp.omroep.nl (145.58.52.132) 56(84) bytes of data.
--- dltest1afp.omroep.nl ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 731ms
rtt min/avg/max/mdev = 87.425/121.739/150.212/17.075 ms, pipe 21
ab isn't happy either:
Transfer rate: 138158.10 [Kbytes/sec] received
and host1 typically shows (sar -n DEV) ~100Mbyte/sec throughput
16:04:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s
txcmp/s rxmcst/s
16:04:27 eth1 37055.00 98042.00 2171.21 144954.92 0.00
0.00 0.00
When testing with lower concurrency (ab -c 8) all is well we typically get
close to 10Gbit throughput.
Transfer rate: 939407.67 [Kbytes/sec] received
So far I've tried different kernel versions (3.0.x, 3.5.x),
different ethtool and sysctl tunings but so far I haven't found
anything to improve performance. Could it perhaps be a driver issue?
Details:
Hardware: IBM HS22 blades with dual 10G mezzanine adapter, connected
via a Cisco Catalyst Switch 3110G for IBM BladeCenter.
Software: We've tested using a vanilla linux 3.5.3 kernel.
smp_affinity follows the hints set by the driver; i.e.
/proc/irq/<NN>/smp_affinity is set to /proc/irq/<NN>/affinity_hint
for the relevant interrupts.
$ dmesg|grep ixgbe
[ 6.509784] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version
3.9.15-k
[ 6.517510] ixgbe: Copyright (c) 1999-2012 Intel Corporation.
[ 6.617763] ixgbe 0000:15:00.0: irq 74 for MSI/MSI-X
[ 6.617770] ixgbe 0000:15:00.0: irq 75 for MSI/MSI-X
[ 6.617776] ixgbe 0000:15:00.0: irq 76 for MSI/MSI-X
[ 6.617782] ixgbe 0000:15:00.0: irq 77 for MSI/MSI-X
[ 6.617789] ixgbe 0000:15:00.0: irq 78 for MSI/MSI-X
[ 6.617795] ixgbe 0000:15:00.0: irq 79 for MSI/MSI-X
[ 6.617801] ixgbe 0000:15:00.0: irq 80 for MSI/MSI-X
[ 6.617807] ixgbe 0000:15:00.0: irq 81 for MSI/MSI-X
[ 6.617813] ixgbe 0000:15:00.0: irq 82 for MSI/MSI-X
[ 6.617819] ixgbe 0000:15:00.0: irq 83 for MSI/MSI-X
[ 6.617825] ixgbe 0000:15:00.0: irq 84 for MSI/MSI-X
[ 6.617830] ixgbe 0000:15:00.0: irq 85 for MSI/MSI-X
[ 6.617836] ixgbe 0000:15:00.0: irq 86 for MSI/MSI-X
[ 6.617842] ixgbe 0000:15:00.0: irq 87 for MSI/MSI-X
[ 6.617848] ixgbe 0000:15:00.0: irq 88 for MSI/MSI-X
[ 6.617854] ixgbe 0000:15:00.0: irq 89 for MSI/MSI-X
[ 6.617860] ixgbe 0000:15:00.0: irq 90 for MSI/MSI-X
[ 6.617910] ixgbe 0000:15:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx
Queue count = 16
[ 6.626456] ixgbe 0000:15:00.0: (PCI Express:5.0GT/s:Width x8)
a0:36:9f:02:94:04
[ 6.633920] ixgbe 0000:15:00.0: MAC: 2, PHY: 1, PBA No: E46189-012
[ 6.643912] ixgbe 0000:15:00.0: Intel(R) 10 Gigabit Network Connection
[ 6.747566] ixgbe 0000:15:00.1: irq 91 for MSI/MSI-X
[ 6.747574] ixgbe 0000:15:00.1: irq 92 for MSI/MSI-X
[ 6.747580] ixgbe 0000:15:00.1: irq 93 for MSI/MSI-X
[ 6.747586] ixgbe 0000:15:00.1: irq 94 for MSI/MSI-X
[ 6.747595] ixgbe 0000:15:00.1: irq 95 for MSI/MSI-X
[ 6.747601] ixgbe 0000:15:00.1: irq 96 for MSI/MSI-X
[ 6.747610] ixgbe 0000:15:00.1: irq 97 for MSI/MSI-X
[ 6.747616] ixgbe 0000:15:00.1: irq 98 for MSI/MSI-X
[ 6.747622] ixgbe 0000:15:00.1: irq 99 for MSI/MSI-X
[ 6.747628] ixgbe 0000:15:00.1: irq 100 for MSI/MSI-X
[ 6.747634] ixgbe 0000:15:00.1: irq 101 for MSI/MSI-X
[ 6.747640] ixgbe 0000:15:00.1: irq 102 for MSI/MSI-X
[ 6.747645] ixgbe 0000:15:00.1: irq 103 for MSI/MSI-X
[ 6.747651] ixgbe 0000:15:00.1: irq 104 for MSI/MSI-X
[ 6.747657] ixgbe 0000:15:00.1: irq 105 for MSI/MSI-X
[ 6.747663] ixgbe 0000:15:00.1: irq 106 for MSI/MSI-X
[ 6.747669] ixgbe 0000:15:00.1: irq 107 for MSI/MSI-X
[ 6.747714] ixgbe 0000:15:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx
Queue count = 16
[ 6.756260] ixgbe 0000:15:00.1: (PCI Express:5.0GT/s:Width x8)
a0:36:9f:02:94:05
[ 6.763723] ixgbe 0000:15:00.1: MAC: 2, PHY: 1, PBA No: E46189-012
[ 6.773610] ixgbe 0000:15:00.1: Intel(R) 10 Gigabit Network Connection
[ 6.780135] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function
Network Driver - version 2.6.0-k
[ 6.789419] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
$ sudo ethtool -i eth1
driver: ixgbe
version: 3.9.15-k
firmware-version: 0x613e0001
bus-info: 0000:15:00.1
$ sudo lspci -vv|grep -A 35 82599EB
15:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit KX4 Network
Connection (rev 01)
Subsystem: Intel Corporation Ethernet Mezzanine Adapter X520-KX4-2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 24
Region 0: Memory at fb000000 (64-bit, prefetchable) [size=8M]
Region 2: I/O ports at 1020 [size=32]
Region 4: Memory at fb804000 (64-bit, prefetchable) [size=16K]
Expansion ROM at 90000000 [disabled] [size=8M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Mask- TabSize=64
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns,
L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Latency L0
<1us, L1 <8us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
Capabilities: [100] Advanced Error Reporting <?>
Capabilities: [140] Device Serial Number 04-94-02-ff-ff-9f-36-a0
Capabilities: [150] #0e
Capabilities: [160] #10
Kernel driver in use: ixgbe
--
15:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit KX4 Network
Connection (rev 01)
Subsystem: Intel Corporation Ethernet Mezzanine Adapter X520-KX4-2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 34
Region 0: Memory at fa800000 (64-bit, prefetchable) [size=8M]
Region 2: I/O ports at 1000 [size=32]
Region 4: Memory at fb800000 (64-bit, prefetchable) [size=16K]
Expansion ROM at 90800000 [disabled] [size=8M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Mask- TabSize=64
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns,
L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+
TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Latency L0
<1us, L1 <8us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
Capabilities: [100] Advanced Error Reporting <?>
Capabilities: [140] Device Serial Number 04-94-02-ff-ff-9f-36-a0
Capabilities: [150] #0e
Capabilities: [160] #10
Kernel driver in use: ixgbe
--
Dick Snippe, internetbeheerder \ fight war
beh...@omroep.nl, +31 35 677 3555 \ not wars
NPO ICT, Sumatralaan 45, 1217 GP Hilversum, NPO Gebouw A
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
