...oh? I thought that FI_TRANSMIT_COMPLETE was the local completion and 
FI_DELIVERY_COMPLETE was the remote completion. What does this mean, then?

        Applies to fi_sendmsg. Indicates that a completion should not be 
generated until the operation has been successfully transmitted and is no 
longer being tracked by the provider.
        Applies to fi_sendmsg. Indicates that a completion should be generated 
when the operation has been processed by the destination.

Anyway, I realize that I'm trying to have my cake and eat it too, but in 
general, I'm looking for:
        1. Blocking send semantics over unconnected endpoints
        2. You get to send again as soon as the buffer is safe to write to 
(currently my use case for the cq),
        3. You also get some kind of event when we're sure the destination 
received the event,
        4. The application doesn't perform extra copy operations on the message 
unless it's completely unavoidable.

This doesn't mean "return from the send call with the buffer safe ASAP," as in 
that case I would just use the memcpy strategy.

So is fi_tinject with FI_INJECT_COMPLETE what I want? It seems that that's 
probably the case, but I do need unconnected endpoints. Since the man pages say 
that it's an "optimized version of fi_tsend" it leads me to believe I cannot 
use it without establishing a connection.


-----Original Message-----
From: Hefty, Sean 
Sent: Wednesday, September 14, 2016 5:14 PM
To: Jeff Hammond <jeff.scie...@gmail.com>
Cc: Smith, Jonathan D <jonathan.d.sm...@intel.com>; 
libfabric-us...@lists.openfabrics.org; ofiwg@lists.openfabrics.org
Subject: RE: [libfabric-users] Two-stage completion

> What if your provider copies to local buffer, then transmits from 
> there? Local completion happens at the rate of memcpy, which is 
> usually faster than RDMA.

FI_TRANSMIT_COMPLETE is not a local completion.

"A completion guarantees that the operation is no longer dependent on the 
fabric or local resources."

I think you're thinking of FI_INJECT_COMPLETE and the fi_inject call, which 
doesn't return until the operation has completed locally.

FI_DELIVERY_COMPLETE adds to transmit complete that the result is visible to 
the application.

Transmit complete likely occurs after the remote NIC acks the message, but 
before the data may have been placed in memory.  Delivery complete likely means 
that the data is in memory.
ofiwg mailing list

Reply via email to