Patrick Geoffray
Thu, 24 Jul 2008 16:01:21 -0700
Hi Jan, Jan Heichler wrote:
1) most applications are latency driven - not bandwidth driven. That means that half bisectional bandwidth is not cutting your application performance down to 50%. For most applications the impact should be less than 5% - for some it is really 0%.
If the app is purely latency driven, bandwidth (link or bisection) is indeed irrelevant. However, don't underestimate the impact of contention on collective communication: once you exceed the internal buffering in the crossbars, you will have back-pressure. Typically, each crossbar port can buffer in the order of 1-10K these days. So, the larger the message size for the collective and the larger the communicator, the greater the need for effective bisection. At this scale (ie 50 nodes), I agree it's not that important, unless you are bandwidth bounded to begin with.
2) Static routing in IB networks limits your bandwidth for many of the possible communication patterns anyway. For completely random communication it was like below 50%. So you buy a IB fabric with full bisectional but can't use it anyway - reducing the bisectional bandwidth is not impacting that much anymore (as far as i understood most whitepapers)
With static routing on Fat Tree or Clos and pseudo-random traffic (ie real world), you waste ~50% of the bisection you have (actually, the more hops the more waste, but it's not linear). So, if you start with half the theoretical bisection, your effective bisection will roughly be a quarter of that.
Patrick _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf