[EMAIL PROTECTED] wrote on Tue, 22 Aug 2006 11:40 -0400:
> What should a BMI method do if the receive is not posted? Wait?
> Cancel with failure?
Something polite, just like MPI more or less. You can complete the
send at the sender but buffer in the sender or receiver. Or you can
keep the send pending until th receiver shows up. We do the latter
for large messages in IB. For small messages (< 8 kB currently),
expected or not, they queue up in the preposted receiver buffers (up
to 20). Too many small messages and the sender will wait.
> I am still trying to have a clear understanding of how a send varies
> from an unexpected send. The receiver will have some number of pre-
> posted, "generic" receives (i.e. can receive from any peer) to catch
> unexpected sends. If the send is _not_ unexpected (i.e. expected),
> then it implies that the receiver will post a receive for the
> expected send (either pre-posted or slightly late due to clock
> drift), no? If not, what am I missing?
In a BMI implementation, very little difference. A "test" on the
receive side only looks at existing connections, while a
"testunexpected" also does the moral equivalent of accept() on the
listening socket.
To a BMI user, an unexpected message always signals the start of a
new transaction, while an expected message continues an existing
transaction for which you've got outstanding state. Phil said in
his CAC paper, "This reduces complexity on the server side because
the server does not have to anticipate buffer use in advance."
> Is the purpose of the RTS/CTS messages then to stall the sending
> until the receiver has posted the receive?
Yes. (I'm just talking about IB again. GM may be similar.)
> If so, would the receiver
> ever send a CTS to indicate that a match is not forthcoming?
No. Not sure how such information would help the sender.
> Or does
> the receiver only send a CTS when the receive is posted?
Yes. With buffer information to enable RDMA Write.
> In the
> latter case, the sender may time out waiting for a CTS and thus
> cancel the send?
Yes. The sender will get a completion with -PVFS_ETIMEDOUT. The
receiver, if it is still around, will see the connection get closed.
A good receiver implementation would forget any outstanding RTS
messages for which the user never posted a receive. (If it happened
to have sent a CTS, it should rewind that state back to "just
posted, never saw RTS" too.) I confess to never having tested this
situation in IB. You might be able to crib from MX's MPID_Cancel,
assuming somebody implemented that well. :)
-- Pete
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
