e1000-developers,
[Cc-ing sparclinux due to the iommu observations..]
I'm looking at an iperf issue running over ixgbe on linux
on a sparc T5-2 platform (64 cpu) where we cannot get to line-speed
(peaks at 3 Gbps on a 10Gbps link) and I'm trying to get to the bottom
of this.
I've run iperf with 8 threads. Observations are
1. lockstat and perf report that iommu->lock is the hot-lock (in a typical
instance, I get about 21M contentions out of 27M acquisitions, 25 us
avg wait time). Even if I fix this issue (see below), I see:
2. ethtool stat: rx_missed_errors and/or rx_no_dma_resources
goes up (even with just one iperf thread).
Disabling IOMMU is not an option on this arch (sun4v).
But I tried a fix to mitigate #1 by breaking up the iommu map/lock
into locks with finer granularity for map-pools, similar to the
design for iommu on powerpc. That fix takes care of the lockstat output,
but it shows lot of latency for packet receive (14 us wait time on
socket lock without RPS, and even with RPS, the rcu lock has a high
wait time), and throughput still cannot go beyond the 3 Gbps limit.
This suggests that #2 needs to be solved first.
I dont think this is a setup issue, though I could be mistaken:
when I boot solaris on the exact same hardware config, I am able to get
a throughput of approx 9.4 Gbps.
There are other things one could do, to ameliorate iommu overhead on this
e.g., keep a cache of premapped buffers for small packets (such as the
TCP ACK, for example) with a configurable threshold defining "small".
But before I go too far into experimenting with those things, I wanted
to check with e1000-devel to see if this just sub-optimal tuning of
the ixgbe driver.
Attached are the output of the commands (listed in the order they appear
below) for a single iperf thread (similar stats issues are there even in
the 1 thread case)
ethtool -i eth1
lspci -vvv -s <bus>
ethtool -k eth1
ethtool -S eth1
Any insights or tuning-recommendations would be appreciated,
--Sowmini
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
