Hi Donald, I ran a small program that uses libpcap to capture promiscuously from one queue on one core, it uses a 1 second interval sigalarm to monitor capture progress and will exit 5 seconds after no packets have been seen (basically a trigger once the source has stopped sending). On exit it prints out the number of packets observed in the run.
I used the latest driver ixgbe-3.4.24, this was without PF_RING. The program simply grabbed the packets and counted bytes, it barely managed 100Megabits/s before it started dropping packets; the following is from a run at 111Megabits/s. I sent 10,000,000 packets form an external test source, packet size was 64 bytes. My app saw 9,562,113 packets Some of The dropped packets are seen with ethtool -S in rx_missed_errors: 48331 .. rx_pkts_nic: 9951669 I used the script in "../scripts/set_irq_affinity.sh" to pin the interrupts to the processor. Here's a screen scrape from 'top' for the process and the interrupt. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3 root 20 0 0 0 0 S 35.2 0.0 13:41.11 ksoftirqd/0 36037 root 20 0 8616 688 588 R 25.1 0.0 0:05.03 count_pcap Here's the log file (ps please do read on, as there's more results after the log). Sep 8 11:01:57 kernel: [260638.894886] Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.24-NAPI Sep 8 11:01:57 kernel: [260638.894893] Copyright (c) 1999-2011 Intel Corporation. Sep 8 11:01:57 kernel: [260638.894966] ixgbe 0000:0e:00.0: PCI INT A -> GSI 38 (level, low) -> IRQ 38 Sep 8 11:01:57 kernel: [260639.020109] ixgbe: Multiple Queue Support Disabled Sep 8 11:01:57 kernel: [260639.020116] ixgbe: 0000:0e:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS. Sep 8 11:01:57 kernel: [260639.020123] ixgbe: 0000:0e:00.0: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:01:57 kernel: [260639.020130] ixgbe: 0000:0e:00.0: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:01:57 kernel: [260639.044593] ixgbe 0000:0e:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:b5:c0:b0 Sep 8 11:01:57 kernel: [260639.044906] ixgbe 0000:0e:00.0: eth4: MAC: 2, PHY: 15, SFP+: 5, PBA No: G18786-001 Sep 8 11:01:57 kernel: [260639.044913] ixgbe 0000:0e:00.0: eth4: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:01:57 kernel: [260639.052005] ixgbe 0000:0e:00.0: eth4: Intel(R) 10 Gigabit Network Connection Sep 8 11:01:57 kernel: [260639.052040] ixgbe 0000:0e:00.1: PCI INT B -> GSI 45 (level, low) -> IRQ 45 Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): carrier is OFF Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): new Ethernet device (driver: 'ixgbe' ifindex: 50) Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): exported as /org/freedesktop/NetworkManager/Devices/48 Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): now managed Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): device state change: 1 -> 2 (reason 2) Sep 8 11:01:57 NetworkManager[2060]: <info> (eth4): bringing up device. Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): preparing device. Sep 8 11:01:58 NetworkManager[2060]: <info> (eth4): deactivating device (reason: 2). Sep 8 11:01:58 kernel: [260639.196124] ADDRCONF(NETDEV_UP): eth4: link is not ready Sep 8 11:01:58 kernel: [260639.235301] ixgbe 0000:0e:00.0: eth4: detected SFP+: 5 Sep 8 11:01:58 kernel: [260640.130940] ixgbe: Multiple Queue Support Disabled Sep 8 11:01:58 kernel: [260640.130948] ixgbe: 0000:0e:00.1: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS. Sep 8 11:01:58 kernel: [260640.130956] ixgbe: 0000:0e:00.1: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:01:58 kernel: [260640.130963] ixgbe: 0000:0e:00.1: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:01:58 kernel: [260640.155466] ixgbe 0000:0e:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:b5:c0:b1 Sep 8 11:01:58 kernel: [260640.155777] ixgbe 0000:0e:00.1: eth5: MAC: 2, PHY: 1, PBA No: G18786-001 Sep 8 11:01:58 kernel: [260640.155785] ixgbe 0000:0e:00.1: eth5: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:01:59 kernel: [260640.162810] ixgbe 0000:0e:00.1: eth5: Intel(R) 10 Gigabit Network Connection Sep 8 11:01:59 kernel: [260640.162852] ixgbe 0000:0f:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40 Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): carrier is OFF Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): new Ethernet device (driver: 'ixgbe' ifindex: 51) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): exported as /org/freedesktop/NetworkManager/Devices/49 Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): now managed Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): device state change: 1 -> 2 (reason 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): bringing up device. Sep 8 11:01:59 kernel: [260640.286831] ixgbe: Multiple Queue Support Disabled Sep 8 11:01:59 kernel: [260640.286837] ixgbe: 0000:0f:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS. Sep 8 11:01:59 kernel: [260640.286843] ixgbe: 0000:0f:00.0: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:01:59 kernel: [260640.286850] ixgbe: 0000:0f:00.0: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:01:59 kernel: [260640.286893] ixgbe 0000:0f:00.0: (unregistered net_device): FCoE offload feature is not available. Disabling FCoE offload feature Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): preparing device. Sep 8 11:01:59 NetworkManager[2060]: <info> (eth5): deactivating device (reason: 2). Sep 8 11:01:59 kernel: [260640.289025] ADDRCONF(NETDEV_UP): eth5: link is not ready Sep 8 11:01:59 kernel: [260640.311585] ixgbe 0000:0f:00.0: (PCI Express:5.0GT/s:Width x8) 00:e0:ed:ff:60:04 Sep 8 11:01:59 kernel: [260640.311678] ixgbe 0000:0f:00.0: eth6: MAC: 2, PHY: 12, SFP+: 5, PBA No: FFFFFF-0FF Sep 8 11:01:59 kernel: [260640.311689] ixgbe 0000:0f:00.0: eth6: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:01:59 kernel: [260640.311710] ixgbe 0000:0f:00.0: eth6: Intel(R) 10 Gigabit Network Connection Sep 8 11:01:59 kernel: [260640.311752] ixgbe 0000:0f:00.1: PCI INT B -> GSI 50 (level, low) -> IRQ 50 Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): carrier is OFF Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): new Ethernet device (driver: 'ixgbe' ifindex: 52) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): exported as /org/freedesktop/NetworkManager/Devices/50 Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): now managed Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 1 -> 2 (reason 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): bringing up device. Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): preparing device. Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): deactivating device (reason: 2). Sep 8 11:01:59 kernel: [260640.451779] ADDRCONF(NETDEV_UP): eth6: link is not ready Sep 8 11:01:59 kernel: [260640.495549] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8 11:01:59 kernel: [260640.640869] ixgbe 0000:0f:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: RX/TX Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): carrier now ON (device state 2) Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 2 -> 3 (reason 40) Sep 8 11:01:59 kernel: [260640.648957] ADDRCONF(NETDEV_CHANGE): eth6: link becomes ready Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) starting connection 'System eth6' Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 3 -> 4 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5 (Device Prepare) scheduled... Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5 (Device Prepare) started... Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) scheduled... Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 1 of 5 (Device Prepare) complete. Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) starting... Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 4 -> 5 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) successful. Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP Configure Start) scheduled. Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 2 of 5 (Device Configure) complete. Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP Configure Start) started... Sep 8 11:01:59 NetworkManager[2060]: <info> (eth6): device state change: 5 -> 7 (reason 0) Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5 (IP4 Configure Get) scheduled... Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 3 of 5 (IP Configure Start) complete. Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5 (IP4 Configure Get) started... Sep 8 11:01:59 NetworkManager[2060]: <info> Scheduling stage 5 Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP Configure Commit) scheduled... Sep 8 11:01:59 NetworkManager[2060]: <info> Done scheduling stage 5 Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 4 of 5 (IP4 Configure Get) complete. Sep 8 11:01:59 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP Configure Commit) started... Sep 8 11:01:59 avahi-daemon[2073]: Joining mDNS multicast group on interface eth6.IPv4 with address 10.0.0.2. Sep 8 11:01:59 avahi-daemon[2073]: New relevant interface eth6.IPv4 for mDNS. Sep 8 11:01:59 avahi-daemon[2073]: Registering new address record for 10.0.0.2 on eth6.IPv4. Sep 8 11:02:00 kernel: [260641.389586] ixgbe: Multiple Queue Support Disabled Sep 8 11:02:00 kernel: [260641.389593] ixgbe: 0000:0f:00.1: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS. Sep 8 11:02:00 kernel: [260641.389600] ixgbe: 0000:0f:00.1: ixgbe_check_options: Flow Director is not supported while multiple queues are disabled. Disabling Flow Director Sep 8 11:02:00 kernel: [260641.389618] ixgbe: 0000:0f:00.1: ixgbe_check_options: FCoE Offload feature enabled Sep 8 11:02:00 kernel: [260641.389661] ixgbe 0000:0f:00.1: (unregistered net_device): FCoE offload feature is not available. Disabling FCoE offload feature Sep 8 11:02:00 kernel: [260641.414340] ixgbe 0000:0f:00.1: (PCI Express:5.0GT/s:Width x8) 00:e0:ed:ff:60:05 Sep 8 11:02:00 kernel: [260641.414432] ixgbe 0000:0f:00.1: eth7: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF Sep 8 11:02:00 kernel: [260641.414443] ixgbe 0000:0f:00.1: eth7: Enabled Features: RxQ: 1 TxQ: 1 RSC Sep 8 11:02:00 kernel: [260641.414464] ixgbe 0000:0f:00.1: eth7: Intel(R) 10 Gigabit Network Connection Sep 8 11:02:00 NetworkManager[2060]: <info> Policy set 'System eth0' (eth0) as default for IPv4 routing and DNS. Sep 8 11:02:00 NetworkManager[2060]: <info> (eth6): device state change: 7 -> 8 (reason 0) Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6) successful, device activated. Sep 8 11:02:00 NetworkManager[2060]: <info> Activation (eth6) Stage 5 of 5 (IP Configure Commit) complete. Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): carrier is OFF Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): new Ethernet device (driver: 'ixgbe' ifindex: 53) Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): exported as /org/freedesktop/NetworkManager/Devices/51 Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): now managed Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): device state change: 1 -> 2 (reason 2) Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): bringing up device. Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): preparing device. Sep 8 11:02:00 NetworkManager[2060]: <info> (eth7): deactivating device (reason: 2). Sep 8 11:02:00 kernel: [260641.769646] ADDRCONF(NETDEV_UP): eth7: link is not ready Sep 8 11:02:01 avahi-daemon[2073]: Registering new address record for fe80::2e0:edff:feff:6004 on eth6.*. Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6): carrier now OFF (device state 8, deferring action for 4 seconds) Sep 8 11:02:05 kernel: [260646.629042] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8 11:02:05 kernel: [260646.851639] ixgbe 0000:0f:00.0: eth6: detected SFP+: 5 Sep 8 11:02:05 kernel: [260647.090225] ixgbe 0000:0f:00.0: eth6: NIC Link is Up 10 Gbps, Flow Control: TX Sep 8 11:02:05 NetworkManager[2060]: <info> (eth6): carrier now ON (device state 8) Sep 8 11:02:16 kernel: [260657.874831] device eth6 entered promiscuous mode Sep 8 11:03:11 kernel: [260712.799174] device eth6 left promiscuous mode Sep 8 11:06:50 kernel: [260930.664527] device eth6 entered promiscuous mode Sep 8 11:07:48 kernel: [260988.578813] device eth6 left promiscuous mode Next I ran the same app but with no options passed to the ixgbe driver, so it defaulted to 64 queues; this is a 32 processor machine with Hyper Threading on. I managed to get to 180 Megabits/s before packets were dropped. Next I turned off Hyper Threading and then ran the driver with the default arguments; this doesn't seem to make any difference, if anything performance was slightly poorer than with it on. Is this what you would have expected for the standard driver? Does it seem like its operating correctly? Andrew ------------------------------------------------------------------------------ Why Cloud-Based Security and Archiving Make Sense Osterman Research conducted this study that outlines how and why cloud computing security and archiving is rapidly being adopted across the IT space for its ease of implementation, lower cost, and increased reliability. Learn more. http://www.accelacomm.com/jaw/sfnl/114/51425301/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
