Patrick Geoffray
Tue, 23 Sep 2008 19:11:43 -0700
Perry E. Metzger wrote:
You realize that most big HPC systems are using interconnects that don't generate many or any interrupts, right?Of course. Usually one even uses interrupt pacing/mitigation even in gig ethernet on a modern machine -- otherwise you're not going to get reasonable performance. (For 10Gig, you have to do even uglier tricks.)
What Greg is trying to say is that high-speed interconnects used in HPC do not raises interrupts at all. Data is delivered directly in user-space, and the app (or the communication library) busy polls on it, no kernel/OS involved. There is one app process per core (usually bound to improve locality in a NUMA architecture). When a daemon wakes up, it will preempt a core, and the app process just has to wait. If the app is tighly coupled, that will delay everybody.
You can say that a daemon waking up every couple of hours is no big deal. However, if these events are uniformly distributed on a couple thousand nodes, it will happen a couple thousand times more often. You can solve this by gang scheduling the daemons across all the nodes, or you can turn them off.
However, it is only important for large machines with tightly coupled codes. For the majority of the cases, it's just being anal.
Patrick _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf