On 24.10.2012 01:13, Garrett Cooper wrote:
On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper <yaneg...@gmail.com> wrote:
Hi,
Doing some poking around at the ixgb driver with a card I have at
$work using netperf and two machines hooked up over crossover, I
discovered that while ixgb's throughput performance was fantastic on
7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1
by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO
performance on the other hand is fantastic and doesn't degrade with
the card across FreeBSD versions. Performance remains constant with
ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage.
More details:
The machines are hooked up in the following configuration:
----------------------- --------------------
| Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 |
----------------------- ---------------------
Machine Configuration:
The card in Machine 2 is an 82599EB card according to pciconf -lv.
/boot/loader.conf tunables (most of these are set according to 9.x
defaults in order to establish a sane baseline):
kern.ipc.nmbjumbo9=262144
kern.ipc.nmbjumbo16=262144
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144
kern.ipc.maxsockbuf=2097152
/etc/sysctl.conf tunables:
net.inet.tcp.recvspace=65536
net.inet.tcp.recvspace_inc=16384
net.inet.tcp.recvspace_max=2097152
net.inet.tcp.sendspace=32768
net.inet.tcp.sendbuf_max=2097152
net.inet.tcp.sendbuf_inc=8192
Kernel Config:
Machine 1 is running a custom version of FreeBSD. The version has been
constant over the course of my testing. Can give vague details on the
config, but can't give some specific details.
Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel.
Networking configuration:
- Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured.
The interface mtu is 1500.
- Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured.
The interface mtu is 1500.
Netperf configuration:
- netserver is run on both machines; I don't add any additional
arguments to the netserver invocation so it just goes off and forks.
- netperf is run like: netperf -cCjt TCP_STREAM -H <IP-ADDRESS>
I was wondering if this was a known issue and/or others had seen
similar problems with this card. I haven't gone into profiling the
kernel yet with DTrace, but if no one gets back to me before sometime
later on this week/next week that will be my next course of action for
tracking down the source of the performance problem.
A couple more notes:
1. We're not using Intel SFP modules -- they're Finisar based
(shouldn't matter, but I know that some Intel NICs are incredibly
picky when it comes to SFP modules).
2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps.
There have been a very large number of changes to our network
stack between 7.4 and today. That makes it very difficult to
pinpoint a specific change.
First of all please test 10-current as well but make sure that
you turn off WITNESS and INVARIANTS in your kernel config.
Second there may be an issue with TSO and ixgb where TSO can
cause packets that are a bit larger than 64K by the ethernet
header size. ixgb DMA is set up for 64K all inclusive. While
TSO did generate a IP valid packet it is arguably not very well
digestable for the driver considering that 64K is a very magic
number. TSO should reduce its largest packet size by max_linkhdr.
A fix for that is in the works and will be available shortly.
--
Andre
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
