I'm not sure this is the correct forum for this problem, but I'll give it a shot anyway. I'm using a very simple UDP benchmark that attempts to send packets as quickly as possible. The platform is a dual Xeon 3.06GHz with HyperThreading, for a total of 4 logical processors, and 4 Intel Gigabit NICs on a PCI 64/66 bus (82546EB). In the experiments, I am pinning four sending processes (one per NIC) to logical processors 0 and 2, and the NIC interrupts to logical processors 1 and 3, such that interrupts are serviced by the sibling logical processor of the one sending the packets. I have used both the driver that comes with the Linux kernel (7.3.20-k2-NAPI), as well as a more recent one (7.6.15.5).
On a vanilla 2.6.25.10 kernel, I can saturate all NICs (4 Gbps), and have some idle CPU cycles to spare. Moving to 2.6.26, with the same configuration file, I get at most 3.4 Gbps, and the sending processors are saturated. All bandwidth numbers were verified on the server and client sides with ethtool statistics. Below are the mpstat averages for a 30 second experiment. There are a few anomalies: 1. The number of IRQs/sec goes down with the default settings. I used InterruptThrottleRate to bring it back to the 2.6.25.10 values, but with no effect on performance. 2. System time for the sending processors goes up in 2.6.26, as well as soft IRQ time for the interrupt-handling processors. 3. In both cases, soft IRQ time is attributed to the sending processors, even though the interrupts are pinned, and soft IRQs should execute on the same processors as the hard IRQs. 4. OProfile results (top 3 entries for each experiment attached below) suggest a huge increase in the time taken by e1000_clean_tx_irq. These should be taken with a grain of salt, as I have lost some confidence in OProfile, especially in a multi-processor, HyperThreaded environment. Any help would be greatly appreciated. Elad == 2.6.25.10 == mpstat: CPU %user %sys %irq %soft %idle intr/s all 0.83 31.74 1.13 23.01 43.20 16017.44 0 1.57 63.85 0.00 15.77 18.81 1.73 1 0.07 0.10 4.50 27.54 67.52 8007.90 2 1.73 62.95 0.00 14.96 20.36 0.00 3 0.00 0.03 0.00 33.74 66.12 8007.90 4 0.00 0.00 0.00 0.00 0.00 0.00 oprofile: 66325 26.4879 e1000.ko e1000 e1000_xmit_frame 46852 18.7111 e1000.ko e1000 e1000_clean_tx_irq 43968 17.5593 e1000.ko e1000 e1000_intr == 2.6.26 == mpstat: CPU %user %sys %irq %soft %idle intr/s all 1.94 36.61 0.81 32.40 28.23 11586.84 0 3.50 73.18 0.00 23.33 0.00 1.73 1 0.63 0.20 3.27 37.03 58.87 5547.32 2 3.43 72.97 0.00 23.60 0.00 1.27 3 0.23 0.10 0.00 45.63 54.03 6036.52 4 0.00 0.00 0.00 0.00 0.00 0.00 oprofile: 235946 52.8313 e1000.ko e1000 e1000_clean_tx_irq 42654 9.5508 e1000.ko e1000 e1000_xmit_frame 40120 8.9834 e1000.ko e1000 e1000_set_mac_type ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel
