On Thu, May 10, 2007 at 04:30:27PM +0300, Or Gerlitz wrote:
> Jeff Squyres wrote:
> >On May 10, 2007, at 9:02 AM, Or Gerlitz wrote:
>
> >>To start with, my hope here is at least to be able play defensive
> >>here, that is convince you that the disadvantages are minor, where
> >>only if this fails, would schedule myself some reading into the
> >>ipoib-cm rfc to dig the advantages.
>
> >I ask about the advantages because OMPI currently treats QP's as
> >bi-directional. Having OMPI treat them at unidirectional would be a
> >change. I'm not against such a change, but I think we'd need to be
> >convinced that there are good reasons to do so. For example, on the
> >surface, it seems like this scheme would simply consume more QPs and
> >potentially more registered memory (and is therefore unattractive).
>
> Indeed you would need two QPs per btl connection, however, for each
> direction you can make the relevant QP consume ~zero resources per the
> other direction, ie on side A:
>
> for the A --> B QP : RX WR num = 0, RX SG size = 0
> for the B --> A QP : TX WR num = 0, TX SG size = 0
>
> and on side B the other way. I think that IB disallows to have zero len
> WR num so you set it actually to 1. Note that since you use SRQ for
> large jobs you have zero overhead for RX resources and this one TX WR
> overhead for the "RX" connection on each side. This is the only memory
> related overhead since you don't have to allocate any extra buffers over
> what you do now.
>
QP is a limited resource and we already have 2 per connection (and much
more if LMC is in used), so I don't see any reason to use this scheme only
to overcome brain damaged design of iWarp.
--
Gleb.
