On May 10, 2007, at 9:02 AM, Or Gerlitz wrote:
A different approach which you might want to consider is to have
at the btl level --two-- connections per <src,dst> ranks. so if A
wants to send B it does so through the A --> B connection and if
B wants to send A it does so through the B --> A connection. To
some extent, this is the approach taken by IPoIB-CM (I am not
enough into the RFC to understand the reasoning but i am quite
sure this was the approach in the initial implementation). At
first thought it mights seems not very elegant, but taking it
into the details (projected on the ompi env) you might find it
even nice.
What is the advantage of this approach?
To start with, my hope here is at least to be able play defensive
here, that is convince you that the disadvantages are minor, where
only if this fails, would schedule myself some reading into the
ipoib-cm rfc to dig the advantages.
I ask about the advantages because OMPI currently treats QP's as bi-
directional. Having OMPI treat them at unidirectional would be a
change. I'm not against such a change, but I think we'd need to be
convinced that there are good reasons to do so. For example, on the
surface, it seems like this scheme would simply consume more QPs and
potentially more registered memory (and is therefore unattractive).
--
Jeff Squyres
Cisco Systems
