>
> The fi_cq(3) man page says that
>
> [FI_DELIVERY_COMPLETE] applies only to reliable endpoints. For
> operations that
> return data to the initiator, such as RMA read or atomic-fetch, the source
> endpoint is
> also considered a destination endpoint. This is the default completion mode
> for such
> operations.
>
>
> Does "default completion mode" in that last sentence really mean "default",
> in that
> it's the completion mode that will be used if and only if no
> FI_{INJECT,TRANSMIT,DELIVERY,...}_COMPLETE flag is present in the hints to
> fi_getinfo()
> and/or the flags to fi_readmsg() etc.? Or does it really mean that
> delivery_complete
> is guaranteed for transactions that produce local results, such that even if
> for
> example FI_TRANSMIT_COMPLETE is given in the hints, RMA reads and fetching
> atomics will
> nevertheless effectively use delivery_complete?
It's intended to mean the former. If the application specifically requests
some other model, then that is all that it app _should_ assume. Note that the
provider can always provide strong guarantees.
> (I'm pretty sure it's the former, but thought I should check.)
>
> If it is indeed the former, when an RMA read or fetching atomic is done and
> FI_TRANSMIT_COMPLETE is in force, is there any way to tell when the data has
> arrived?
This is probably where we can argue that delivery_complete is the minimal
semantic that makes sense for RMA read or fetching atomic.
A more technical answer is that a fenced operation can be used to ensure that
all prior operations meet the same completion semantic as the fenced op. So,
if a fenced op has deliver_complete, then the completion of the fenced op means
that operations prior to that also meet that semantic.
I believe existing hardware provides delivery_complete semantics. I hesitate
to make this a requirement in case someone designs a NIC with more exotic
caching capabilities, or if the target memory is non-host memory.
- Sean
_______________________________________________
ofiwg mailing list
[email protected]
https://lists.openfabrics.org/mailman/listinfo/ofiwg
