Hello, Todd Fujinaka & List
Am Montag, 16. März 2015 17:56:08 schrieb Fujinaka, Todd:
> These cards don't have firmware. They're just configuration settings. You
> can download the OpenSDM on the files section to decode the differences,
> but mostly it's identification information.
Great.
So we can safely rule the "firmware" item out of the list of suspects :-)
I kept moving things around, and now, on top of my list of possible
bottlenecks is the mainboard.
When I change the position of the NICs on the board, the bottleneck stays with
the PCIe socket, not with the NIC or the external network path.
============
my results in more detail:
I figured out that I can not only put one single iperf on my aggregated link,
but still also access the individual links, since teql is a layer 3
aggregation scheme, and all patways still have their own IP and MAC.
This way I also can rule out possible problems caused by the teql layer.
So I did
for i in 62 94 126 158 190 222; do ( iperf -c 192.168.130.$i -t1 & ) ; done
to call a number of iperf instances in parallel, each over a different GBit
link.
On a "good link" I get > 980 MBit / s out of 1000 MBit possible.
On a "bad link", I get < 800 MBit / s
I can have all 4 Links on the "good NIC" and 1 link on the "bad NIC" at close
to 1000 MBIt.
A teql aggregation over those five links gets iperf > 4.7 GBit/s out of 5
Gbit, which is close to 95 % - I'd say: "perfect".
However, when I run two paralle iperf over the "bad NIC", bandwith gets down
to 740 Mbit/s for each. Even worse, teql seems to be irritated by the
reschedule / reordering, so aggregated bandwith goes down even more. But this
is a knwon issue in link aggregation, so I really were desperate to work on
top of "clean" physical links.
Next, I interchanged external connections, but the problem stuck with the
NICs, so I can rule out external causes, like cabling, switches, VLAN setup,
client system.
Then I interchanged the PCIe slot position of the NICs in the mainboard, and
now I see that the "bad NIC" is still the one in PCI bus position 07:00 /
08:00. So it is not a question of NIC Branding or a hardware failure there.
I can establish the following combined iperf connection rates on "bad" NIC /
slot:
1000 Mbit/s
730 + 730 Mbit/s
730 + 400 + 400 MBit/s
400 + 400 + 400 + 400 MBit/s
Those rates are reproducible at +- 20 MBit and independent on the traffic
generated on the other NIC (which may be from 0 to 4x 1Gbit/s).
Up to now, I tried debug level 4 and 7 on loading module e1000e, but I still
get no trace of packet loss in my syslog.
Or do I still have to enable somthing?
Like some secret sysfs paramter?
Where to go from here?
update of Possible causes on my list
### - firmware problem (NICs)
!!!! firmware problem ( Mainboard)
###- hardware problem (NICs,
!!! hardware Mainboard)
!!!! conceptual limitation of hardware design
???? driver problem
???? kernel / scheduling issue / IRQ / race...whatever?
???? some realy weird hidden tweak paramater
#### still the nasty VC blade switch?
( which led me already to migrate from layer 2 bonding to layer 3 teql)
#### other network / client side issues
???? any more????
### ruled out
??? still under consideration
!!! top suspect
So Ill gooogle for sabertooth, but there are still some ?? on driver related
issues.
Wolfgang Rosner
==============================================
more details on Mainboard
(from hwinfo ...)
type 0x02 [0x0002]: 02 0f 02 00 01 02 03 04 05 09 06 03 00 0a 00
str1: "ASUSTeK COMPUTER INC."
str2: "SABERTOOTH 990FX R2.0"
str3: "Rev 1.xx"
str4: "141033355400133"
root@cruncher:/cluster/etc/scripts/available# lspci | grep -i ether
07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
0c:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0c:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0d:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0d:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0e:05.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 10)
10:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
PCI Express Gigabit Ethernet Controller (rev 09)
root@cruncher:/cluster/etc/scripts/available# lspci -t
-[0000:00]-+-00.0
+-02.0-[01]--+-00.0
| \-00.1
+-04.0-[02]----00.0
+-05.0-[03]----00.0
+-09.0-[04]----00.0
+-0a.0-[05-08]----00.0-[06-08]--+-00.0-[07]--+-00.0
| | \-00.1
| \-01.0-[08]--+-00.0
| \-00.1
+-0b.0-[09]--+-00.0
| \-00.1
+-0d.0-[0a-0d]----00.0-[0b-0d]--+-02.0-[0c]--+-00.0
| | \-00.1
| \-04.0-[0d]--+-00.0
| \-00.1
+-11.0
+-12.0
+-12.2
+-13.0
+-13.2
+-14.0
+-14.2
+-14.3
+-14.4-[0e]----05.0
+-14.5
+-15.0-[0f]--
+-15.1-[10]----00.0
+-15.2-[11]----00.0
+-15.3-[12]----00.0
+-16.0
+-16.2
+-18.0
+-18.1
+-18.2
+-18.3
+-18.4
\-18.5
root@cruncher:/cluster/etc/scripts/available# lspci -vvs 00:0a
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx1 port A) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
I/O behind bridge: 00007000-00008fff
Memory behind bridge: fc500000-fc6fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #5, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit
Latency L0s <1us, L1 <8us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
Surprise-
Slot #10, PowerLimit 75.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power-
Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+
Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible+
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-,
OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-,
OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [b0] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI]
Device 5a14
Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1
Len=010 <?>
Capabilities: [190 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl- DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd-
EgressCtrl- DirectTrans-
Kernel driver in use: pcieport
root@cruncher:/cluster/etc/scripts/available# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx0 port B) (rev 02)
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port B)
00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port D)
00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port E)
00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port H)
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx1 port A)
00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (NB-SB link)
00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx1 port B)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller
(rev 42)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia
(Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI
Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900
PCI to PCI bridge (PCIE port 0)
00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900
PCI to PCI bridge (PCIE port 1)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI
bridge (PCIE port 2)
00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI
bridge (PCIE port 3)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 5
01:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
01:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio
Controller (rev a1)
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller
(rev 01)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller
(rev 01)
04:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host
Controller
05:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI
Express Switch (rev 04)
06:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI
Express Switch (rev 04)
06:01.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI
Express Switch (rev 04)
07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
08:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
09:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000]
(rev a1)
09:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio
Controller (rev a1)
0a:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI
Express Switch (rev 0e)
0b:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI
Express Switch (rev 0e)
0b:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI
Express Switch (rev 0e)
0c:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0c:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0d:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0d:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) (rev 06)
0e:05.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 10)
10:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
PCI Express Gigabit Ethernet Controller (rev 09)
11:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host
Controller
12:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host
Controller
>
> Todd Fujinaka
> Software Application Engineer
> Networking Division (ND)
> Intel Corporation
> todd.fujin...@intel.com
> (503) 712-4565
>
> -----Original Message-----
> From: Wolfgang Rosner [mailto:wros...@tirnet.de]
> Sent: Monday, March 16, 2015 9:43 AM
> To: Scott Silverman
> Cc: e1000-devel@lists.sourceforge.net
> Subject: Re: [E1000-devel] rx_missed_errors with e1000e on 82571EB chip
> only on HP, not on Intel card
>
> Am Montag, 16. März 2015 14:14:06 schrieben Sie:
> > Probably unrelated,
>
> surely not!
>
> > but I remember that this was once a thing:
> > http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c015015
> > 74
>......
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
