Alternatively, is there anything in the SCST layer I should tweak. I'm
still running rev 245 of that code (kinda old, but works with OFED 1.3.1
w/o hacks).
And sorry about the yucky signature on my past email - I used to be able
to send w/o that.
Cameron Harr wrote:
Greetings,
While trying to maximize IOPs over SRP, I ran into a pretty
significant bottleneck. After digging into the matter, I believe the
problem is coming from the mlx4 drivers overwhelming one of the cpus,
and thus cutting performance to that level (since no more IB traffic
means no more data for the disks).
My target server (with DAS) contains 8 2.8 GHz CPU cores and can
sustain over 200K IOPs locally, but only around 73K IOPs over SRP.
Looking at /proc/interrupts, I see that the mlx_core (comp) device is
pushing about 135K Int/s on 1 of 2 CPUs. All CPUs are enabled for that
PCI-E slot, but it only ever uses 2 of the CPUs, and only 1 at a time.
None of the other CPUs has an interrupt rate more than about 40-50K/s.
Does anyone know of a trick to spread those interrupts out more (which
I realize might be bad due to context switching), or something else
that will reduce my interrupts on that cpu? The mlx4 is a MSI-X
interrupt. I've changed it to an APIC int, but it seems to give
slightly lower performance.
Thanks,
Cameron
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
