Jeff Squyres wrote:
On Jul 12, 2007, at 1:18 PM, Don Kerr wrote:
- So if you want to simply eliminate the flow control, choose M high
enough (or just a total number of receive buffers to post to the SRQ)
that you won't ever run out of resources and you should see some
speedup from lack of flow control. This obviously mainly helps apps
with lots of small messages; it may not help in many other cases.
Is there any distinction by the size of the message. If the "M"
parameter is set high does the openib btl post this many recv buffers
for the SRQ on both QPs? Or are SRQs only created on one of the QPs?
Keep in mind that the SRQs are only for send/receive messages, not
RDMA messages.
That is obviously enough but isn't there a window for MPI messages that
are greater than the eager limit but less than where the rdma protocol
kicks in and fragments for this size message use fragments larger than
than the eager size.
Maybe this is where openib's high and low priority qp differ from udapl
which makes a choice of which endpoint to use based on the size of the
fragment. That is why I was curious if openib was using SRQs on both
queue pairs.
Each receive buffer has a max size (the eager limit, IIRC). So if
the message is larger than that, we'll fragment per the pipeline
protocol, possibly subject to doing RDMA if the message is large
enough, yadda yadda yadda. More specifically, the size of the buffer
is not dependent upon an individual message that is being sent or
received (since they're pre-posted -- we have no idea what the
message sizes will be).
As for whether the SRQ is on both QP's, this is a Galen/George/Gleb
(G^3) question...
