Hi Donald,
I removed the second 10 Gig NIC and switched off the second pair of built-in
1 Gig via the BIOS so as to reduce the bus loading, as a result I now have 4
NIC active. Unfortunately, this change didn't seem to make much difference to
performance.
Next I reduced the number of active cores (processors) in the system, again
using the BIOS settings, and switched Hyper Threading off. Here's the results
of 10,000,000 packets 64 bytes per packet, listed is the highest throughput
before packets were dropped.
1 Queue num proc Queues
4 procs 110Mbits/s 125Mbits/s
8 procs 110Mbits/s 215Mbits/s
16 procs 110 Mbits/s 180Mbits/s
When running with "num proc" queues I simply used the default driver values
with i.e. "insmod ixgbe.ko", so that the number of queues matched the number of
cores. I used the ixgbe-3.4.24 driver.
I did notice that on 16 cores when the packets started dropping, the CPU usage
of the processes labeled ksoftirqd/proc_num i.e. "ksoftirqd/0, ksoftirqd/1 and
so on through 15" when through the roof so that before packets were dropped
those processes barely registered in 'top' but as soon as packets were dropped
these processes dominated the output of top, almost like they were thrashing.
In a simple experiment to prove this, I noted that in the case of 16 processors
at 175Mbits/s these processes were under 1% CPU (mostly 0.0%) use but at, say
190Mbits/s these processing dominated the output of top, most at > 90% with
some in the mid 50%; it was almost like they were thrashing.
Again I am seeing the kind of throughput I was before, I guess this is much
less than you'd expect?
Hope this helps...
Andrew
-----Original Message-----
From: andrew_leh...@agilent.com [mailto:andrew_leh...@agilent.com]
Sent: Thursday, September 08, 2011 3:47 PM
To: donald.c.skidm...@intel.com; d...@ntop.org
Cc: e1000-devel@lists.sourceforge.net
Subject: Re: [E1000-devel] Problems with Dell R810.
Hi Donald,
I ran a small program that uses libpcap to capture promiscuously from one
queue on one core, it uses a 1 second interval sigalarm to monitor capture
progress and will exit 5 seconds after no packets have been seen (basically a
trigger once the source has stopped sending). On exit it prints out the number
of packets observed in the run.
I used the latest driver ixgbe-3.4.24, this was without PF_RING. The program
simply grabbed the packets and counted bytes, it barely managed 100Megabits/s
before it started dropping packets; the following is from a run at
111Megabits/s.
I sent 10,000,000 packets form an external test source, packet size was 64
bytes.
My app saw 9,562,113 packets
Some of The dropped packets are seen with ethtool -S in
rx_missed_errors: 48331
..
rx_pkts_nic: 9951669
I used the script in "../scripts/set_irq_affinity.sh" to pin the interrupts to
the processor.
Here's a screen scrape from 'top' for the process and the interrupt.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3 root 20 0 0 0 0 S 35.2 0.0 13:41.11 ksoftirqd/0
36037 root 20 0 8616 688 588 R 25.1 0.0 0:05.03 count_pcap
Here's the log file (ps please do read on, as there's more results after the
log).
Sep 8 11:01:57 kernel: [260638.894886] Intel(R) 10 Gigabit PCI Express Network
Driver - version 3.4.24-NAPI Sep 8 11:01:57 kernel: [260638.894893] Copyright
(c) 1999-2011 Intel Corporation.
Sep 8 11:01:57 kernel: [260638.894966] ixgbe 0000:0e:00.0: PCI INT A -> GSI 38
(level, low) -> IRQ 38 Sep 8 11:01:57 kernel: [260639.020109] ixgbe: Multiple
Queue Support Disabled Sep 8 11:01:57 kernel: [260639.020116] ixgbe:
0000:0e:00.0: ixgbe_check_options: RSS is not supported while multiple queues
are disabled. Disabling RSS.
Sep 8 11:01:57 kernel: [260639.020123] ixgbe: 0000:0e:00.0:
ixgbe_check_options: Flow Director is not supported while multiple queues are
disabled. Disabling Flow Director Sep 8 11:01:57 kernel: [260639.020130]
ixgbe: 0000:0e:00.0: ixgbe_check_options: FCoE Offload feature enabled Sep 8
11:01:57 kernel: [260639.044593] ixgbe 0000:0e:00.0: (PCI Express:5.0GT/s:Width
x8) 00:1b:21:b5:c0:b0 Sep 8 11:01:57 kernel: [260639.044906] ixgbe
0000:0e:00.0: eth4: MAC: 2, PHY: 15, SFP+: 5, PBA No: G18786-001 Sep 8
11:01:57 kernel: [260639.044913] ixgbe 0000:0e:00.0: eth4: Enabled Features:
RxQ: 1 TxQ: 1 RSC Sep 8 11:01:57 kernel: [260639.052005] ixgbe 0000:0e:00.0:
eth4: Intel(R) 10 Gigabit Network Connection Sep 8 11:01:57 kernel:
[260639.052040] ixgbe 0000:0e:00.1: PCI INT B -> GSI 45 (level, low) -> IRQ 45
Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): carrier is OFF Sep 8
11:01:57 NetworkManager[2060]: <info> (eth4): new Ethernet device (driver:
'ixgbe' ifindex: 50) Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4):
exported as /org/freedesktop/NetworkManager/Devices/48
Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): now managed Sep 8
11:01:57 NetworkManager[2060]: <info> (eth4): device state change: 1 -> 2
(reason 2) Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): bringing up
device.
Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): preparing device.
Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): deactivating device
(reason: 2).
Sep 8 11:01:58 kernel: [260639.196124] ADDRCONF(NETDEV_UP): eth4: link is not
ready Sep 8 11:01:58 kernel: [260639.235301] ixgbe 0000:0e:00.0: eth4:
detected SFP+: 5 Sep 8 11:01:58 kernel: [260640.130940] ixgbe: Multiple Queue
Support Disabled Sep 8 11:01:58 kernel: [260640.130948] ixgbe: 0000:0e:00.1:
ixgbe_check_options: RSS is not supported while multiple queues are disabled.
Disabling RSS.
Sep 8 11:01:58 kernel: [260640.130956] ixgbe: 0000:0e:00.1:
ixgbe_check_options: Flow Director is not supported while multiple queues are
disabled. Disabling Flow Director Sep 8 11:01:58 kernel: [260640.130963]
ixgbe: 0000:0e:00.1: ixgbe_check_options: FCoE Offload feature enabled Sep 8
11:01:58 kernel: [260640.155466] ixgbe 0000:0e:00.1: (PCI Express:5.0GT/s:Width
x8) 00:1b:21:b5:c0:b1 Sep 8 11:01:58 kernel: [260640.155777] ixgbe
0000:0e:00.1: eth5: MAC: 2, PHY: 1, PBA No: G18786-001 Sep 8 11:01:58 kernel:
[260640.155785] ixgbe 0000:0e:00.1: eth5: Enabled Features: RxQ: 1 TxQ: 1 RSC
Sep 8 11:01:59 kernel: [260640.162810] ixgbe 0000:0e:00.1: eth5: Intel(R) 10
Gigabit Network Connection Sep 8 11:01:59 kernel: [260640.162852] ixgbe
0000:0f:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40 Sep 8 11:01:59
NetworkManager[2060]: <info> (eth5): carrier is OFF Sep 8 11:01:59
NetworkManager[2060]: <info> (eth5): new Ethernet device (driver: 'ixgbe'
ifindex: 51) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): exported as
/org/freedesktop/NetworkManager/Devices/49
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): now managed Sep 8
11:01:59 NetworkManager[2060]: <info> (eth5): device state change: 1 -> 2
(reason 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): bringing up
device.
Sep 8 11:01:59 kernel: [260640.286831] ixgbe: Multiple Queue Support Disabled
Sep 8 11:01:59 kernel: [260640.286837] ixgbe: 0000:0f:00.0:
ixgbe_check_options: RSS is not supported while multiple queues are disabled.
Disabling RSS.
Sep 8 11:01:59 kernel: [260640.286843] ixgbe: 0000:0f:00.0:
ixgbe_check_options: Flow Director is not supported while multiple queues are
disabled. Disabling Flow Director Sep 8 11:01:59 kernel: [260640.286850]
ixgbe: 0000:0f:00.0: ixgbe_check_options: FCoE Offload feature enabled Sep 8
11:01:59 kernel: [260640.286893] ixgbe 0000:0f:00.0: (unregistered net_device):
FCoE offload feature is not available. Disabling FCoE offload feature Sep 8
11:01:59 NetworkManager[2060]: <info> (eth5): preparing device.
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): deactivating device
(reason: 2).
Sep 8 11:01:59 kernel: [260640.289025] ADDRCONF(NETDEV_UP): eth5: link is not
ready Sep 8 11:01:59 kernel: [260640.311585] ixgbe 0000:0f:00.0: (PCI
Express:5.0GT/s:Width x8) 00:e0:ed:ff:60:04 Sep 8 11:01:59 kernel:
[260640.311678] ixgbe 0000:0f:00.0: eth6: MAC: 2, PHY: 12, SFP+: 5, PBA No:
FFFFFF-0FF Sep 8 11:01:59 kernel: [260640.311689] ixgbe 0000:0f:00.0: eth6:
Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:01:59 kernel: [260640.311710]
ixgbe 0000:0f:00.0: eth6: Intel(R) 10 Gigabit Network Connection Sep 8
11:01:59 kernel: [260640.311752] ixgbe 0000:0f:00.1: PCI INT B -> GSI 50
(level, low) -> IRQ 50 Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6):
carrier is OFF Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): new
Ethernet device (driver: 'ixgbe' ifindex: 52) Sep 8 11:01:59
NetworkManager[2060]: <info> (eth6): exported as
/org/freedesktop/NetworkManager/Devices/50
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): now managed Sep 8
11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 1 -> 2
(reason 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): bringing up
device.
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): preparing device.
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): deactivating device
(reason: 2).
Sep 8 11:01:59 kernel: [260640.451779] ADDRCONF(NETDEV_UP): eth6: link is not
ready Sep 8 11:01:59 kernel: [260640.495549] ixgbe 0000:0f:00.0: eth6:
detected SFP+: 5 Sep 8 11:01:59 kernel: [260640.640869] ixgbe 0000:0f:00.0:
eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX Sep 8 11:01:59
NetworkManager[2060]: <info> (eth6): carrier now ON (device state 2) Sep 8
11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 2 -> 3
(reason 40) Sep 8 11:01:59 kernel: [260640.648957] ADDRCONF(NETDEV_CHANGE):
eth6: link becomes ready Sep 8 11:01:59 NetworkManager[2060]: <info>
Activation (eth6) starting connection 'System eth6'
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 3 ->
4 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6)
Stage 1 of 5 (Device Prepare) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5
(Device Prepare) started...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5
(Device Configure) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5
(Device Prepare) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5
(Device Configure) starting...
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 4 ->
5 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6)
Stage 2 of 5 (Device Configure) successful.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP
Configure Start) scheduled.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5
(Device Configure) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP
Configure Start) started...
Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 5 ->
7 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6)
Stage 4 of 5 (IP4 Configure Get) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP
Configure Start) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5
(IP4 Configure Get) started...
Sep 8 11:01:59 NetworkManager[2060]: <info> Scheduling stage 5 Sep 8 11:01:59
NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP Configure
Commit) scheduled...
Sep 8 11:01:59 NetworkManager[2060]: <info> Done scheduling stage 5 Sep 8
11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5 (IP4
Configure Get) complete.
Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP
Configure Commit) started...
Sep 8 11:01:59 avahi-daemon[2073]: Joining mDNS multicast group on interface
eth6.IPv4 with address 10.0.0.2.
Sep 8 11:01:59 avahi-daemon[2073]: New relevant interface eth6.IPv4 for mDNS.
Sep 8 11:01:59 avahi-daemon[2073]: Registering new address record for 10.0.0.2
on eth6.IPv4.
Sep 8 11:02:00 kernel: [260641.389586] ixgbe: Multiple Queue Support Disabled
Sep 8 11:02:00 kernel: [260641.389593] ixgbe: 0000:0f:00.1:
ixgbe_check_options: RSS is not supported while multiple queues are disabled.
Disabling RSS.
Sep 8 11:02:00 kernel: [260641.389600] ixgbe: 0000:0f:00.1:
ixgbe_check_options: Flow Director is not supported while multiple queues are
disabled. Disabling Flow Director Sep 8 11:02:00 kernel: [260641.389618]
ixgbe: 0000:0f:00.1: ixgbe_check_options: FCoE Offload feature enabled Sep 8
11:02:00 kernel: [260641.389661] ixgbe 0000:0f:00.1: (unregistered net_device):
FCoE offload feature is not available. Disabling FCoE offload feature Sep 8
11:02:00 kernel: [260641.414340] ixgbe 0000:0f:00.1: (PCI Express:5.0GT/s:Width
x8) 00:e0:ed:ff:60:05 Sep 8 11:02:00 kernel: [260641.414432] ixgbe
0000:0f:00.1: eth7: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF Sep 8 11:02:00 kernel:
[260641.414443] ixgbe 0000:0f:00.1: eth7: Enabled Features: RxQ: 1 TxQ: 1 RSC
Sep 8 11:02:00 kernel: [260641.414464] ixgbe 0000:0f:00.1: eth7: Intel(R) 10
Gigabit Network Connection Sep 8 11:02:00 NetworkManager[2060]: <info> Policy
set 'System eth0' (eth0) as default for IPv4 routing and DNS.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth6): device state change: 7 ->
8 (reason 0) Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6)
successful, device activated.
Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP
Configure Commit) complete.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): carrier is OFF Sep 8
11:02:00 NetworkManager[2060]: <info> (eth7): new Ethernet device (driver:
'ixgbe' ifindex: 53) Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7):
exported as /org/freedesktop/NetworkManager/Devices/51
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): now managed Sep 8
11:02:00 NetworkManager[2060]: <info> (eth7): device state change: 1 -> 2
(reason 2) Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): bringing up
device.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): preparing device.
Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): deactivating device
(reason: 2).
Sep 8 11:02:00 kernel: [260641.769646] ADDRCONF(NETDEV_UP): eth7: link is not
ready Sep 8 11:02:01 avahi-daemon[2073]: Registering new address record for
fe80::2e0:edff:feff:6004 on eth6.*.
Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6): carrier now OFF (device
state 8, deferring action for 4 seconds) Sep 8 11:02:05 kernel:
[260646.629042] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8 11:02:05
kernel: [260646.851639] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8
11:02:05 kernel: [260647.090225] ixgbe 0000:0f:00.0: eth6: NIC Link is Up 10
Gbps, Flow Control: TX Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6):
carrier now ON (device state 8) Sep 8 11:02:16 kernel: [260657.874831] device
eth6 entered promiscuous mode Sep 8 11:03:11 kernel: [260712.799174] device
eth6 left promiscuous mode Sep 8 11:06:50 kernel: [260930.664527] device eth6
entered promiscuous mode Sep 8 11:07:48 kernel: [260988.578813] device eth6
left promiscuous mode
Next I ran the same app but with no options passed to the ixgbe driver, so it
defaulted to 64 queues; this is a 32 processor machine with Hyper Threading on.
I managed to get to 180 Megabits/s before packets were dropped.
Next I turned off Hyper Threading and then ran the driver with the default
arguments; this doesn't seem to make any difference, if anything performance
was slightly poorer than with it on.
Is this what you would have expected for the standard driver? Does it seem like
its operating correctly?
Andrew
------------------------------------------------------------------------------
Why Cloud-Based Security and Archiving Make Sense Osterman Research conducted
this study that outlines how and why cloud computing security and archiving is
rapidly being adopted across the IT space for its ease of implementation, lower
cost, and increased reliability. Learn more.
http://www.accelacomm.com/jaw/sfnl/114/51425301/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
------------------------------------------------------------------------------
BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
Learn about the latest advances in developing for the
BlackBerry® mobile platform with sessions, labs & more.
See new tools and technologies. Register for BlackBerry® DevCon today!
http://p.sf.net/sfu/rim-devcon-copy1
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
