...oh? I thought that FI_TRANSMIT_COMPLETE was the local completion and
FI_DELIVERY_COMPLETE was the remote completion. What does this mean, then?
FI_TRANSMIT_COMPLETE
Applies to fi_sendmsg. Indicates that a completion should not be
generated until the operation has been successfully transmitted and is no
longer being tracked by the provider.
FI_DELIVERY_COMPLETE
Applies to fi_sendmsg. Indicates that a completion should be generated
when the operation has been processed by the destination.
Anyway, I realize that I'm trying to have my cake and eat it too, but in
general, I'm looking for:
1. Blocking send semantics over unconnected endpoints
2. You get to send again as soon as the buffer is safe to write to
(currently my use case for the cq),
3. You also get some kind of event when we're sure the destination
received the event,
4. The application doesn't perform extra copy operations on the message
unless it's completely unavoidable.
This doesn't mean "return from the send call with the buffer safe ASAP," as in
that case I would just use the memcpy strategy.
So is fi_tinject with FI_INJECT_COMPLETE what I want? It seems that that's
probably the case, but I do need unconnected endpoints. Since the man pages say
that it's an "optimized version of fi_tsend" it leads me to believe I cannot
use it without establishing a connection.
Thanks,
Jonathan
-----Original Message-----
From: Hefty, Sean
Sent: Wednesday, September 14, 2016 5:14 PM
To: Jeff Hammond <jeff.scie...@gmail.com>
Cc: Smith, Jonathan D <jonathan.d.sm...@intel.com>;
libfabric-us...@lists.openfabrics.org; ofiwg@lists.openfabrics.org
Subject: RE: [libfabric-users] Two-stage completion
> What if your provider copies to local buffer, then transmits from
> there? Local completion happens at the rate of memcpy, which is
> usually faster than RDMA.
FI_TRANSMIT_COMPLETE is not a local completion.
"A completion guarantees that the operation is no longer dependent on the
fabric or local resources."
I think you're thinking of FI_INJECT_COMPLETE and the fi_inject call, which
doesn't return until the operation has completed locally.
FI_DELIVERY_COMPLETE adds to transmit complete that the result is visible to
the application.
Transmit complete likely occurs after the remote NIC acks the message, but
before the data may have been placed in memory. Delivery complete likely means
that the data is in memory.
_______________________________________________
ofiwg mailing list
ofiwg@lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/ofiwg
