Hi,
I'm running a workload on a T5440 2-node cluster with 2 x 1 GBit interconnects
which is saturating the cpus processing the interrupts for incoming network
packets (Solaris 10 U5). The system has 4 onboard Neptune 1 GBit NICs and 2
QuadPort 1 GBit NICs (all nxge). Currently, I only get 2 interrupt cpus per
port:
# echo ::interrupts | mdb -k | grep nxge
nxge#1 no MSI-X 11 enbl 0x23 0x523 6 179
nxge#1 no MSI-X 10 enbl 0x22 0x522 6 178
...
nxge#9 no MSI-X 9 enbl 0x20 0x620 6 187
nxge#9 no MSI-X 8 enbl 0x1f 0x61f 6 186
Those four cpus are around 100% utilized (mpstat), resulting in bad network
latencies.
I know that 2 is the default and that by setting ddi_msix_alloc_limit this may
be increased. Afaik those Quad-Port boards have 16 receive channels in total
and should allow for up to 4 interrupt cpus per port. But setting
ddi_msix_alloc_limit to 4 doesn't help (I'm still getting just 2 cpus). A quick
look at the source code seems to indicate that the number of interrupt cpus is
limited to 2 for 1G NICs (and 8 for 10G NICs):
nxge_main.c
370 /* PSARC/2007/453 MSI-X interrupt limit override. */
371 #define NXGE_MSIX_REQUEST_10G 8
372 #define NXGE_MSIX_REQUEST_1G 2
Is there any way to get more than 2 interrupt cpus for 1G NICs?
If not, why is the number limited to 2, although we have 16 receive channels on
a quad-port 1G card?
Yes, I could switch to 10G NICs (which I don't want) or use 4 physical links as
interconnects (which I also do not want). Probably fencing those interrupt cpus
into a processor set would help a little, if there were no application threads
running on these cpus, but this is also no solution for a product system.
What about the project "Interrupt Resource Management"? Will that change
anything? Or the polling mechanism in Crossbow? I assume, both have not yet
made their way into S10U5?
Thanks for your help,
Nick.
--
This message posted from opensolaris.org
_______________________________________________
networking-discuss mailing list
[email protected]
