Hello Alex,
> Specifically the 2.5GT/s with a width of
> x1 can barely push 1Gb/s. This slot needs to be at least a x4 if you
> want to push anything more than 1Gb/s.
Thank you for the hint.
I think this was the major issue.
I've only been looking to the LnkSta of the NIC itself, where it reported
LnkSta 2.5GT/s, Width x4.
I didn't expect that there was such kind of a bottleneck in the "uplink" path.
I have filed a support enquiry with asus on the case.
For the quick hack, I switched PCI slots of the "starved" NIC with a Graphic
card. The board is advertised as a Gamer's toy, with up to 4 Graphic cards,
thats why I expected to get ample of PCIe bandwith out of it.
For the moment it works OK for me, although it still does not look like things
behaving to spec.
The vendors ad says:
"Die PCIe-2.0-x16-Slots laufen mit 16/16/0/4 bzw. 16/8/8/4 Lanes."
which translates to
PCIe-2.0-x16-Slots are runing on 16/16/0/4 or 16/8/8/4 Lanes, respectively.
hm. well, respecting to what? To the number of cards present, I suppose?
So even with all 4 PCIe-x16 slots in use, there should not be a fallback to
x1, should it?
But maybe it is not the manufacturers fault, but a linux driver problem?
Well, but even if so, I'm already on the wrong list again, I'm afraid.
And why do we only have 2.5 GT/s
According to
http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_2.0
PCIe-2.0 is supposed to deliver 5 GT.
ah, I see, the NICs say
LnkCap: Port #0, Speed 2.5GT/s, Width x4,
so they are not capable of PCIe-2.0, right?
Are there some knobs and screws where I can fine tune the PCIe configuration,
or do I have to rely on the black magic of kernel & mainboard?
===============
all right, current perfomance tests look like this:
iperf client -> server over 6 pyhsical Links :
6 x 990 MBit - OK
iperf client -> server over 1 teql Aggregate :
5700 MBit > 90 % - OK
iperf server - client over 6 phys links
6 x 990...1000 MBIt -OK
iperf client -> server over 1 teql Aggregate :
~ 3000 MBit ~~ 50 % - ##### NOT OK #####
iperf client -> server over 10 parallel teql Aggregate :
~ 5800 MBit > 95 % - OK
iperf client -> server over 2 parallel teql Aggregate :
~ 5900 MBit > 95 % - OK
So I think the link layer is fine now, and I have no sign of loosing packets.
But there still seems to be some send bottleneck related to the the singe TCP
channel in the case wit one aggregated link.
What is the first parameter to look for?
I'm quite sure it is among the list of things I tested the last days, wich did
not solve the PCIe bottleneck.
Wolfgang Rosner
Am Mittwoch, 18. März 2015 21:22:52 schrieben Sie:
> On 03/18/2015 01:04 PM, Wolfgang Rosner wrote:
> > root@cruncher:/cluster/etc/scripts/available# lspci -vvs 00:0a
> > 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to
> > PCI bridge (external gfx1 port A) (prog-if 00 [Normal decode])
.......
> > LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
> > DLActive+ BWMgmt+ ABWMgmt-
> > SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> > Surprise-
>
> I think this is your problem. Specifically the 2.5GT/s with a width of
> x1 can barely push 1Gb/s. This slot needs to be at least a x4 if you
> want to push anything more than 1Gb/s.
>
> - Alex
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
