Hey Andrew, Thanks for the detailed description. A couple things of note:
- When you're only using one queue ATF won't be enabled since there is only one queue to route all flows will be routed to it anyway. So the set_irq_affinity.sh script is pretty much useless here since it affinitizies the queues to cpu based on cpu number. - If you're seeing rx_missed_errors this means that the receive FIFO has insufficient space to store incoming packets. Normally this is due to insufficient bandwidth on the IO bus. To answer your question this is not the performance I would expect. From the dmesg it looks like you have a few Ethernet devices on this system could they be filling up your bus? I would also be interested in seeing your interrupt spread (proc/interrupts) when running with multiple queues. Likewise the whole output of "ethtool -S" to see if anything else pops out. As for your testing environment are you running back-2-back with your traffic generator or threw a switch? Also what are you using to generate your traffic, pktgen? Thanks, -Don >-----Original Message----- >From: andrew_leh...@agilent.com [mailto:andrew_leh...@agilent.com] >Sent: Thursday, September 08, 2011 7:47 AM >To: Skidmore, Donald C; d...@ntop.org >Cc: e1000-devel@lists.sourceforge.net >Subject: RE: Problems with Dell R810. > >Hi Donald, > > I ran a small program that uses libpcap to capture promiscuously from >one queue on one core, it uses a 1 second interval sigalarm to monitor >capture progress and will exit 5 seconds after no packets have been seen >(basically a trigger once the source has stopped sending). On exit it >prints out the number of packets observed in the run. > >I used the latest driver ixgbe-3.4.24, this was without PF_RING. The >program simply grabbed the packets and counted bytes, it barely managed >100Megabits/s before it started dropping packets; the following is from >a run at 111Megabits/s. > >I sent 10,000,000 packets form an external test source, packet size was >64 bytes. > >My app saw 9,562,113 packets > >Some of The dropped packets are seen with ethtool -S in > >rx_missed_errors: 48331 >.. >rx_pkts_nic: 9951669 > >I used the script in "../scripts/set_irq_affinity.sh" to pin the >interrupts to the processor. > >Here's a screen scrape from 'top' for the process and the interrupt. > >PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >3 root 20 0 0 0 0 S 35.2 0.0 13:41.11 ksoftirqd/0 >36037 root 20 0 8616 688 588 R 25.1 0.0 0:05.03 count_pcap > > >Here's the log file (ps please do read on, as there's more results after >the log). > >Sep 8 11:01:57 kernel: [260638.894886] Intel(R) 10 Gigabit PCI Express >Network Driver - version 3.4.24-NAPI >Sep 8 11:01:57 kernel: [260638.894893] Copyright (c) 1999-2011 Intel >Corporation. >Sep 8 11:01:57 kernel: [260638.894966] ixgbe 0000:0e:00.0: PCI INT A -> >GSI 38 (level, low) -> IRQ 38 >Sep 8 11:01:57 kernel: [260639.020109] ixgbe: Multiple Queue Support >Disabled >Sep 8 11:01:57 kernel: [260639.020116] ixgbe: 0000:0e:00.0: >ixgbe_check_options: RSS is not supported while multiple queues are >disabled. Disabling RSS. >Sep 8 11:01:57 kernel: [260639.020123] ixgbe: 0000:0e:00.0: >ixgbe_check_options: Flow Director is not supported while multiple >queues are disabled. Disabling Flow Director >Sep 8 11:01:57 kernel: [260639.020130] ixgbe: 0000:0e:00.0: >ixgbe_check_options: FCoE Offload feature enabled >Sep 8 11:01:57 kernel: [260639.044593] ixgbe 0000:0e:00.0: (PCI >Express:5.0GT/s:Width x8) 00:1b:21:b5:c0:b0 >Sep 8 11:01:57 kernel: [260639.044906] ixgbe 0000:0e:00.0: eth4: MAC: >2, PHY: 15, SFP+: 5, PBA No: G18786-001 >Sep 8 11:01:57 kernel: [260639.044913] ixgbe 0000:0e:00.0: eth4: >Enabled Features: RxQ: 1 TxQ: 1 RSC >Sep 8 11:01:57 kernel: [260639.052005] ixgbe 0000:0e:00.0: eth4: >Intel(R) 10 Gigabit Network Connection >Sep 8 11:01:57 kernel: [260639.052040] ixgbe 0000:0e:00.1: PCI INT B -> >GSI 45 (level, low) -> IRQ 45 >Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): carrier is OFF >Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): new Ethernet device >(driver: 'ixgbe' ifindex: 50) >Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): exported as >/org/freedesktop/NetworkManager/Devices/48 >Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): now managed >Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): device state >change: 1 -> 2 (reason 2) >Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): bringing up device. >Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): preparing device. >Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): deactivating device >(reason: 2). >Sep 8 11:01:58 kernel: [260639.196124] ADDRCONF(NETDEV_UP): eth4: link >is not ready >Sep 8 11:01:58 kernel: [260639.235301] ixgbe 0000:0e:00.0: eth4: >detected SFP+: 5 >Sep 8 11:01:58 kernel: [260640.130940] ixgbe: Multiple Queue Support >Disabled >Sep 8 11:01:58 kernel: [260640.130948] ixgbe: 0000:0e:00.1: >ixgbe_check_options: RSS is not supported while multiple queues are >disabled. Disabling RSS. >Sep 8 11:01:58 kernel: [260640.130956] ixgbe: 0000:0e:00.1: >ixgbe_check_options: Flow Director is not supported while multiple >queues are disabled. Disabling Flow Director >Sep 8 11:01:58 kernel: [260640.130963] ixgbe: 0000:0e:00.1: >ixgbe_check_options: FCoE Offload feature enabled >Sep 8 11:01:58 kernel: [260640.155466] ixgbe 0000:0e:00.1: (PCI >Express:5.0GT/s:Width x8) 00:1b:21:b5:c0:b1 >Sep 8 11:01:58 kernel: [260640.155777] ixgbe 0000:0e:00.1: eth5: MAC: >2, PHY: 1, PBA No: G18786-001 >Sep 8 11:01:58 kernel: [260640.155785] ixgbe 0000:0e:00.1: eth5: >Enabled Features: RxQ: 1 TxQ: 1 RSC >Sep 8 11:01:59 kernel: [260640.162810] ixgbe 0000:0e:00.1: eth5: >Intel(R) 10 Gigabit Network Connection >Sep 8 11:01:59 kernel: [260640.162852] ixgbe 0000:0f:00.0: PCI INT A -> >GSI 40 (level, low) -> IRQ 40 >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): carrier is OFF >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): new Ethernet device >(driver: 'ixgbe' ifindex: 51) >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): exported as >/org/freedesktop/NetworkManager/Devices/49 >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): now managed >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): device state >change: 1 -> 2 (reason 2) >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): bringing up device. >Sep 8 11:01:59 kernel: [260640.286831] ixgbe: Multiple Queue Support >Disabled >Sep 8 11:01:59 kernel: [260640.286837] ixgbe: 0000:0f:00.0: >ixgbe_check_options: RSS is not supported while multiple queues are >disabled. Disabling RSS. >Sep 8 11:01:59 kernel: [260640.286843] ixgbe: 0000:0f:00.0: >ixgbe_check_options: Flow Director is not supported while multiple >queues are disabled. Disabling Flow Director >Sep 8 11:01:59 kernel: [260640.286850] ixgbe: 0000:0f:00.0: >ixgbe_check_options: FCoE Offload feature enabled >Sep 8 11:01:59 kernel: [260640.286893] ixgbe 0000:0f:00.0: >(unregistered net_device): FCoE offload feature is not available. >Disabling FCoE offload feature >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): preparing device. >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): deactivating device >(reason: 2). >Sep 8 11:01:59 kernel: [260640.289025] ADDRCONF(NETDEV_UP): eth5: link >is not ready >Sep 8 11:01:59 kernel: [260640.311585] ixgbe 0000:0f:00.0: (PCI >Express:5.0GT/s:Width x8) 00:e0:ed:ff:60:04 >Sep 8 11:01:59 kernel: [260640.311678] ixgbe 0000:0f:00.0: eth6: MAC: >2, PHY: 12, SFP+: 5, PBA No: FFFFFF-0FF >Sep 8 11:01:59 kernel: [260640.311689] ixgbe 0000:0f:00.0: eth6: >Enabled Features: RxQ: 1 TxQ: 1 RSC >Sep 8 11:01:59 kernel: [260640.311710] ixgbe 0000:0f:00.0: eth6: >Intel(R) 10 Gigabit Network Connection >Sep 8 11:01:59 kernel: [260640.311752] ixgbe 0000:0f:00.1: PCI INT B -> >GSI 50 (level, low) -> IRQ 50 >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): carrier is OFF >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): new Ethernet device >(driver: 'ixgbe' ifindex: 52) >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): exported as >/org/freedesktop/NetworkManager/Devices/50 >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): now managed >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state >change: 1 -> 2 (reason 2) >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): bringing up device. >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): preparing device. >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): deactivating device >(reason: 2). >Sep 8 11:01:59 kernel: [260640.451779] ADDRCONF(NETDEV_UP): eth6: link >is not ready >Sep 8 11:01:59 kernel: [260640.495549] ixgbe 0000:0f:00.0: eth6: >detected SFP+: 5 >Sep 8 11:01:59 kernel: [260640.640869] ixgbe 0000:0f:00.0: eth6: NIC >Link is Up 10 Gbps, Flow Control: RX/TX >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): carrier now ON >(device state 2) >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state >change: 2 -> 3 (reason 40) >Sep 8 11:01:59 kernel: [260640.648957] ADDRCONF(NETDEV_CHANGE): eth6: >link becomes ready >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) starting >connection 'System eth6' >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state >change: 3 -> 4 (reason 0) >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 >of 5 (Device Prepare) scheduled... >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 >of 5 (Device Prepare) started... >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 >of 5 (Device Configure) scheduled... >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 >of 5 (Device Prepare) complete. >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 >of 5 (Device Configure) starting... >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state >change: 4 -> 5 (reason 0) >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 >of 5 (Device Configure) successful. >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 >of 5 (IP Configure Start) scheduled. >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 >of 5 (Device Configure) complete. >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 >of 5 (IP Configure Start) started... >Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state >change: 5 -> 7 (reason 0) >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 >of 5 (IP4 Configure Get) scheduled... >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 >of 5 (IP Configure Start) complete. >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 >of 5 (IP4 Configure Get) started... >Sep 8 11:01:59 NetworkManager[2060]: <info> Scheduling stage 5 >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 5 >of 5 (IP Configure Commit) scheduled... >Sep 8 11:01:59 NetworkManager[2060]: <info> Done scheduling stage 5 >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 >of 5 (IP4 Configure Get) complete. >Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 5 >of 5 (IP Configure Commit) started... >Sep 8 11:01:59 avahi-daemon[2073]: Joining mDNS multicast group on >interface eth6.IPv4 with address 10.0.0.2. >Sep 8 11:01:59 avahi-daemon[2073]: New relevant interface eth6.IPv4 for >mDNS. >Sep 8 11:01:59 avahi-daemon[2073]: Registering new address record for >10.0.0.2 on eth6.IPv4. >Sep 8 11:02:00 kernel: [260641.389586] ixgbe: Multiple Queue Support >Disabled >Sep 8 11:02:00 kernel: [260641.389593] ixgbe: 0000:0f:00.1: >ixgbe_check_options: RSS is not supported while multiple queues are >disabled. Disabling RSS. >Sep 8 11:02:00 kernel: [260641.389600] ixgbe: 0000:0f:00.1: >ixgbe_check_options: Flow Director is not supported while multiple >queues are disabled. Disabling Flow Director >Sep 8 11:02:00 kernel: [260641.389618] ixgbe: 0000:0f:00.1: >ixgbe_check_options: FCoE Offload feature enabled >Sep 8 11:02:00 kernel: [260641.389661] ixgbe 0000:0f:00.1: >(unregistered net_device): FCoE offload feature is not available. >Disabling FCoE offload feature >Sep 8 11:02:00 kernel: [260641.414340] ixgbe 0000:0f:00.1: (PCI >Express:5.0GT/s:Width x8) 00:e0:ed:ff:60:05 >Sep 8 11:02:00 kernel: [260641.414432] ixgbe 0000:0f:00.1: eth7: MAC: >2, PHY: 1, PBA No: FFFFFF-0FF >Sep 8 11:02:00 kernel: [260641.414443] ixgbe 0000:0f:00.1: eth7: >Enabled Features: RxQ: 1 TxQ: 1 RSC >Sep 8 11:02:00 kernel: [260641.414464] ixgbe 0000:0f:00.1: eth7: >Intel(R) 10 Gigabit Network Connection >Sep 8 11:02:00 NetworkManager[2060]: <info> Policy set 'System eth0' >(eth0) as default for IPv4 routing and DNS. >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth6): device state >change: 7 -> 8 (reason 0) >Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6) >successful, device activated. >Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6) Stage 5 >of 5 (IP Configure Commit) complete. >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): carrier is OFF >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): new Ethernet device >(driver: 'ixgbe' ifindex: 53) >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): exported as >/org/freedesktop/NetworkManager/Devices/51 >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): now managed >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): device state >change: 1 -> 2 (reason 2) >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): bringing up device. >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): preparing device. >Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): deactivating device >(reason: 2). >Sep 8 11:02:00 kernel: [260641.769646] ADDRCONF(NETDEV_UP): eth7: link >is not ready >Sep 8 11:02:01 avahi-daemon[2073]: Registering new address record for >fe80::2e0:edff:feff:6004 on eth6.*. >Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6): carrier now OFF >(device state 8, deferring action for 4 seconds) >Sep 8 11:02:05 kernel: [260646.629042] ixgbe 0000:0f:00.0: eth6: >detected SFP+: 5 >Sep 8 11:02:05 kernel: [260646.851639] ixgbe 0000:0f:00.0: eth6: >detected SFP+: 5 >Sep 8 11:02:05 kernel: [260647.090225] ixgbe 0000:0f:00.0: eth6: NIC >Link is Up 10 Gbps, Flow Control: TX >Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6): carrier now ON >(device state 8) >Sep 8 11:02:16 kernel: [260657.874831] device eth6 entered promiscuous >mode >Sep 8 11:03:11 kernel: [260712.799174] device eth6 left promiscuous >mode >Sep 8 11:06:50 kernel: [260930.664527] device eth6 entered promiscuous >mode >Sep 8 11:07:48 kernel: [260988.578813] device eth6 left promiscuous >mode > >Next I ran the same app but with no options passed to the ixgbe driver, >so it defaulted to 64 queues; this is a 32 processor machine with Hyper >Threading on. > >I managed to get to 180 Megabits/s before packets were dropped. > >Next I turned off Hyper Threading and then ran the driver with the >default arguments; this doesn't seem to make any difference, if anything >performance was slightly poorer than with it on. > >Is this what you would have expected for the standard driver? Does it >seem like its operating correctly? > >Andrew ------------------------------------------------------------------------------ Why Cloud-Based Security and Archiving Make Sense Osterman Research conducted this study that outlines how and why cloud computing security and archiving is rapidly being adopted across the IT space for its ease of implementation, lower cost, and increased reliability. Learn more. http://www.accelacomm.com/jaw/sfnl/114/51425301/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
