Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-16 Thread Smith, Jonathan D
Hi Sean! Thanks for your help so far. I think I'm getting somewhere! I have some responses and more details for you: > those flags only apply to the fi_sendmsg call. Other send operations do not > take flags, and they do not apply to the fi_recvmsg call. Understood. I'm using connectionless

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-16 Thread Smith, Jonathan D
And then what completion mode should I select? FI_DELIVERY_COMPLETE? -Original Message- From: Hefty, Sean Sent: Thursday, September 15, 2016 3:36 PM To: Smith, Jonathan D ; Jeff Hammond Cc: libfabric-us...@lists.openfabrics.org;

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-16 Thread Hefty, Sean
> I'm not sure that using fi_tsendmsg with the FI_INJECT flag would meet > Jonathan's requirements - if I'm reading this email chain correctly. > I'll reorder his requirements and add comments: > > > 4. The application doesn't perform extra copy operations on the > message unless it's

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-15 Thread Hefty, Sean
> And then what completion mode should I select? FI_DELIVERY_COMPLETE? I would let the provider select this. FI_DELIVERY_COMPLETE can have significant overhead based on the provider implementation. ___ ofiwg mailing list ofiwg@lists.openfabrics.org

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-15 Thread Hefty, Sean
> If I start using the inject call to block until the buffer is safe, how > do I get the kind of completion I'd need for my timeout, if there's no > flags argument in the fi_tinject call? To reuse the buffer immediately but still get a completion, you should call fi_tsendmsg with the FI_INJECT

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-15 Thread Hefty, Sean
> You looking at the flags discussion for the send/receive operations. You -> you're > This is calling out that those flags only apply to the fi_sendmsg call. > Other send operations do no take flags, and they do not apply to the No -> not Apparently I need more sleep.

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-15 Thread Hefty, Sean
> ...oh? I thought that FI_TRANSMIT_COMPLETE was the local completion and > FI_DELIVERY_COMPLETE was the remote completion. What does this mean, > then? > > FI_TRANSMIT_COMPLETE > Applies to fi_sendmsg. Indicates that a completion should not be > generated until the operation has been

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-14 Thread Hefty, Sean
> What if your provider copies to local buffer, then transmits from > there? Local completion happens at the rate of memcpy, which is usually > faster than RDMA. FI_TRANSMIT_COMPLETE is not a local completion. "A completion guarantees that the operation is no longer dependent on the fabric or

Re: [ofiwg] [libfabric-users] Two-stage completion

2016-09-14 Thread Jeff Hammond
On Sep 14, 2016, at 4:59 PM, Hefty, Sean wrote: >> In my application, I could benefit greatly if I could force generation >> of two completion events per fi_tsendmsg – one generated on >> FI_TRANSMIT_COMPLETE and the other on FI_DELIVERY_COMPLETE, so I can >> block until