Scott Atchley <[email protected]> writes: > When I test Open-MX, I turn interrupt coalescing off. I run > omx_pingpong to determine the lowest latency (LL). If the NIC's driver > allows one to specify the interrupt value, I set it to LL-1.
Right, and that's what I did before, with sensible results I thought. Repeating it now on Centos 5.2 and OpenSuSE 10.3, it doesn't behave sensibly, and I don't know what's different from the previous SuSE results apart, probably, from the minor kernel version. If I set rx-frames=0, I see this: rx-usec latency (µs) 20 34.6 12 26.3 6 20.0 1 14.8 whereas if I just set rx-frames=1, I get 14.7 µs, roughly independently of rx-usec. (Those figures are probably ±∼0.2µs.) > If the > driver does not allow specifying the actual rate (i.e. it only has > predetermined values), then I leave it off. Right. (Adaptive coalescence gave significantly higher latency with our nVidia and Intel NICs.) For others interested, this affects TCP results similarly to open-mx, though the base TCP latency is substantially worse, of course. I was going to write this up for the OMX FAQ, but was loath to without understanding the tg3 situation. > The downside is lower throughput for large messages on 10G Ethernet. I > don't think it matters on gigabit. It doesn't affect the ping-pong throughput significantly, but I don't know if it has any effect on the system overall (other cores servicing the interrupts) on `typical' jobs. > Brice and Nathalie have a paper which implements an adaptive interrupt > coalescing so that you do not have to manually tune anything: Isn't that only relevant if you control the firmware? I previously didn't really care about free firmware for devices in the same way as free software generally, but am beginning to see reasons to care. _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
