We have servers that have both IB and ethernet-only clients. The
ethernet-only clients access the servers via IPoIB through a
gateway. For small MTUs, the IpoIB software interrupt load can get
pretty unmanageable so we would like to be able to spread the load
over multiple cores. Our current /proc/interrupts looks like...
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4
CPU5 CPU6 CPU7
snip...
67: 946 0 0 0 1364412993
0 0 0 PCI-MSI-X ib_mthca (comp)
75: 1785 0 0 0 6286882
0 0 0 PCI-MSI-X ib_mthca (async)
snip...
We'd like to be able to spread the irq 67 interrupts across several or
all CPUs but
echo f0 > /proc/irq/67/smp_affinity
seems to have no effect. We can move the interrupts to any
*single* cpu we like but we cannot distribute them among multiple
cpus. The results is that under heavy load from the ethernet-only
clients, the IPoIB traffic becomes interrupt bound.
We found a site (http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
)
that suggests that disabling CONFIG_HOTPLUG_CPU will allow /proc/irq/
nn/smp_affinity to work as expected. My efforts so far with the
2.6.18-92 (RH 5.2) and the 2.6.18-128 kernels have resulted in kernels
that don't boot (after simply configuring CONFIG_HOTPLUG_CPU=n).
I've googled around a bit but have not come up with much. I
thought I'd ask here since we are specifically interested in being
able to do this for our IB HCAs.
Is this a known issue and is there a way to make it work?
Thanks,
Charlie Taylor
UF HPC Center
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
