On Oct 21, 2009, at 3:32 PM, Brice Goglin wrote:
George Bosilca wrote:
On Oct 21, 2009, at 13:42 , Scott Atchley wrote:
On Oct 21, 2009, at 1:25 PM, George Bosilca wrote:
Because MX doesn't provide a real RMA protocol, we created a fake
one on top of point-to-point. The two peers have to agree on a
unique tag, then the receiver posts it before the sender starts the
send. However, as this is integrated with the real RMA protocol,
where only one side knows about the completion of the RMA
operation,
we still exchange the ACK at the end. Therefore, the receiver
doesn't need to know when the receive is completed, as it will get
an ACK from the sender. At least this was the original idea.
But I can see how this might fails if the short ACK from the sender
manage to pass the RMA operation on the wire. I was under the
impression (based on the fact that MX respect the ordering) that
the
mx_send will trigger the completion only when all data is on the
wire/nic memory so I supposed there is _absolutely_ no way for the
ACK to bypass the last RMA fragments and to reach the receiver
before the recv is really completed. If my supposition is not
correct, then we should remove the mx_forget and make sure the that
before we mark a fragment as completed we got both completions (the
one from mx_recv and the remote one).
When is the ACK sent? After the "PUT" completion returns (via
mx_test(), etc) or simply after calling mx_isend() for the "PUT" but
before the completion?
The ACK is sent by the PML layer. If I'm not mistaken, it is sent
when
the completion callback is triggered, which should happen only when
the MX BTL detect the completion of the mx_isend (using the mx_test).
Therefore, I think the ACK is sent in response to the completion of
the mx_isend.
Before or after mx_test() doesn't actually matter if it's a
small/medium. Even if the send(PUT) completes in mx_test(), the data
could still be on the wire in case of packet loss or so: if it's a
tiny/small/medium message (it's was a medium in my crash), the MX lib
opportunistically completes the request on the sender before it's
actually acked by the receiver. Matching is in order, request
completion
is not. There's no strong delivery guarantee here.
Brice
Yes, I was thinking of the rendezvous case (>32 kB) only.
Scott
