Did you ever hear anything back on this? I've been digging a bit today and ran across this:
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/nxge/nxge_main.c#52 52 /* 53 * Software workaround for the hardware 54 * checksum bugs that affect packet transmission 55 * and receive: 56 * 57 * Usage of nxge_cksum_offload: 58 * 59 * (1) nxge_cksum_offload = 0 (default): 60 * - transmits packets: 61 * TCP: uses the hardware checksum feature. 62 * UDP: driver will compute the software checksum 63 * based on the partial checksum computed 64 * by the IP layer. 65 * - receives packets 66 * TCP: marks packets checksum flags based on hardware result. 67 * UDP: will not mark checksum flags. It looks as though if you are doing a lot of UDP traffic software checksums could be contributing to the CPU load. Mike On Wed, Nov 12, 2008 at 12:32 PM, Nicolas Michael <[email protected]> wrote: > Hi, > > I'm running a workload on a T5440 2-node cluster with 2 x 1 GBit > interconnects which is saturating the cpus processing the interrupts for > incoming network packets (Solaris 10 U5). The system has 4 onboard Neptune 1 > GBit NICs and 2 QuadPort 1 GBit NICs (all nxge). Currently, I only get 2 > interrupt cpus per port: > > # echo ::interrupts | mdb -k | grep nxge > nxge#1 no MSI-X 11 enbl 0x23 0x523 6 179 > nxge#1 no MSI-X 10 enbl 0x22 0x522 6 178 > ... > nxge#9 no MSI-X 9 enbl 0x20 0x620 6 187 > nxge#9 no MSI-X 8 enbl 0x1f 0x61f 6 186 > > Those four cpus are around 100% utilized (mpstat), resulting in bad network > latencies. > > I know that 2 is the default and that by setting ddi_msix_alloc_limit this > may be increased. Afaik those Quad-Port boards have 16 receive channels in > total and should allow for up to 4 interrupt cpus per port. But setting > ddi_msix_alloc_limit to 4 doesn't help (I'm still getting just 2 cpus). A > quick look at the source code seems to indicate that the number of interrupt > cpus is limited to 2 for 1G NICs (and 8 for 10G NICs): > > nxge_main.c > 370 /* PSARC/2007/453 MSI-X interrupt limit override. */ > 371 #define NXGE_MSIX_REQUEST_10G 8 > 372 #define NXGE_MSIX_REQUEST_1G 2 > > Is there any way to get more than 2 interrupt cpus for 1G NICs? > If not, why is the number limited to 2, although we have 16 receive channels > on a quad-port 1G card? > > Yes, I could switch to 10G NICs (which I don't want) or use 4 physical links > as interconnects (which I also do not want). Probably fencing those interrupt > cpus into a processor set would help a little, if there were no application > threads running on these cpus, but this is also no solution for a product > system. > > What about the project "Interrupt Resource Management"? Will that change > anything? Or the polling mechanism in Crossbow? I assume, both have not yet > made their way into S10U5? > > Thanks for your help, > Nick. > -- > This message posted from opensolaris.org > _______________________________________________ > networking-discuss mailing list > [email protected] > -- Mike Gerdts http://mgerdts.blogspot.com/ _______________________________________________ networking-discuss mailing list [email protected]
