Dave, Scott,
Dave Love wrote:
Scott Atchley <[email protected]> writes:
When I test Open-MX, I turn interrupt coalescing off. I run
omx_pingpong to determine the lowest latency (LL). If the NIC's driver
allows one to specify the interrupt value, I set it to LL-1.
Note that it is only meaningful wrt ping-pong latency. To optimize for
all latency cases, you just want interrupt coalescing to be off.
results apart, probably, from the minor kernel version. If I set
rx-frames=0, I see this:
rx-usec latency (µs)
20 34.6
12 26.3
6 20.0
1 14.8
whereas if I just set rx-frames=1, I get 14.7 µs, roughly independently
of rx-usec. (Those figures are probably ±∼0.2µs.)
rx-usecs specifies the minimum time between interrupts, whereas
rx-frames specifies the number of frames (packets) between interrupts.
So, if you set rx-frames to 1, there will be an interrupt after each
packet. Not many devices implement rx-frames, since it does not
distinguish between small and large frames. Adaptive coalescing methods
do look at the size of the frames to figure out if the traffic is mostly
latency or bandwidth sensitive, but it's just a guess.
The downside is lower throughput for large messages on 10G Ethernet. I
don't think it matters on gigabit.
It doesn't affect the ping-pong throughput significantly, but I don't
know if it has any effect on the system overall (other cores servicing
the interrupts) on `typical' jobs.
On GigE, each 1500 Bytes frames takes more than 10us on the wire so even
with interrupt coalescing turned off, you won't get more than 100K
interrupts per second. It used to be a problem, but it's no big deal on
recent machines. However, you can get a lot more interrupt when
receiving smaller packets, although the interrupt overhead itself would
limit the interrupt load to well below 1 Million per second. In the
worst case, you would lose a core if you don't let the OS move the
interrupt handler to do load balancing. What is one core these days ? :-)
Patrick
_______________________________________________
Beowulf mailing list, [email protected] sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
