I'm not sure that using fi_tsendmsg with the FI_INJECT flag would meet 
Jonathan's requirements - if I'm reading this email chain correctly. I'll 
reorder his requirements and add comments:

>       4. The application doesn't perform extra copy operations on the message 
> unless it's completely unavoidable.

You also don't want the *provider* to do any internal memory copies. In my 
mind, using FI_INJECT means "release ownership of the source buffer and return 
from this function ASAP". Maybe the provider doesn't have hardware support for 
this and needs to memcpy, or maybe the hardware is busy and the provider needs 
to memcpy. Also, maybe the provider doesn't support a large enough "inject 
size" transmit attribute - the application would have to memcpy. Or maybe the 
provider implements an arbitrarily large "inject size" using and internal 

>       1. Blocking send semantics over unconnected endpoints

This is a blocking send from the perspective of his wrapper library, right?  
The library  implementation would be to spin on fi_cq_read() until a 
FI_INJECT_COMPLETE event and then return to the application.

>       2. You get to send again as soon as the buffer is safe to write to 
> (currently my use case for the cq),

Same as waiting for FI_INJECT_COMPLETE event.

>       3. You also get some kind of event when we're sure the destination 
> received the event


The real question is if two events can be generated for one operation. The 
documentation for the FI_*_COMPLETE flags in the fi_endpoint man page all start 
like this:

        "Indicates that a completion should be generated when ..."

"... a completion ..." sounds like there could be more than one completion 
event. It should say "... the completion ..." if only one event can be 
generated for each operation.

Earlier, Sean said "The timing difference between the two completion models 
seems minimal, especially compared to the time needed to generate, transfer, 
and process an additional ack." But, this is comparing FI_TRANSMIT_COMPLETE and 
FI_DELIVERY_COMPLETE.  The timing difference is quite noticeable between 
FI_INJECT_COMPLETE (local operation) and 
FI_TRANSMIT_COMPLETE|FI_DELIVERY_COMPLETE (remote operation). As Jeff Hammond 
pointed out, several different middleware could use - needs - this distinction 
for best performance.

ALTERNATIVELY, could Jonathan rely on the FI_ORDER_WAS and issue a local atomic 
operation (to increment a counter) after the send operation? Or could he rely 
on FI_ORDER_SAS and issue a "throw away" send using FI_INJECT_COMPLETE in order 
to get the local completion event (the data movement send would use 

Sorry for the long email !


-----Original Message-----
From: Libfabric-users [mailto:libfabric-users-boun...@lists.openfabrics.org] On 
Behalf Of Hefty, Sean
Sent: Thursday, September 15, 2016 5:36 PM
To: Smith, Jonathan D <jonathan.d.sm...@intel.com>; Jeff Hammond 
Cc: ofiwg@lists.openfabrics.org; libfabric-us...@lists.openfabrics.org
Subject: Re: [libfabric-users] Two-stage completion

> If I start using the inject call to block until the buffer is safe, how
> do I get the kind of completion I'd need for my timeout, if there's no
> flags argument in the fi_tinject call?

To reuse the buffer immediately but still get a completion, you should call 
fi_tsendmsg with the FI_INJECT flag.  The default completion mode will then be 
Libfabric-users mailing list
ofiwg mailing list

Reply via email to