Nathan,
Indeed the original design allowed for multiple usages of the same
descriptor, not concurrent as the text in the btl.h indicates but
consecutive. The MCA_BTL_FLAGS_RDMA_MATCHED flag is a weirdness needed for
Portal, and I am not use it is currently in use anywhere in the code base.
My problem with the depicted approach is that now we have two critical
sections in the fast path: one to allocate/reserve the descriptor (this is
at the BTL level on a call from the PML), and then another one to allocate
whatever structure the BTL needs to store the callback informations (again
on a call from the PML to the BTL). In the previous design, we carefully
analyzed all communications path and tried to minimize the number of
back-and-forth between the PML and BTL layer in order to preserve the
performance.
George.
On Thu, Jul 10, 2014 at 2:57 PM, Nathan Hjelm <hje...@lanl.gov> wrote:
>
> What: Change the descriptor completion callback function to include
> cbdata and context pointers.
>
> Old callback:
>
> typedef void (*mca_btl_base_completion_fn_t)(
> struct mca_btl_base_module_t* module,
> struct mca_btl_base_endpoint_t* endpoint,
> struct mca_btl_base_descriptor_t* descriptor,
> int status);
>
>
> New callback:
>
> typedef void (*mca_btl_base_completion_fn_t)(
> struct mca_btl_base_module_t* module,
> struct mca_btl_base_endpoint_t* endpoint,
> struct mca_btl_base_descriptor_t* descriptor,
> void *cbdata, void *context, int status);
>
>
> Why: The BTL interface provides support for using a single descriptor
> with multiple concurrent RDMA operations. BTLs support this feature if
> the following flag is not set:
>
> /** RDMA put/get calls must have a matching prepare_{src,dst} call
> on the target with the same base (and possibly bound). */
> #define MCA_BTL_FLAGS_RDMA_MATCHED 0x0040
>
>
> The problem is that in order to pass back the correct callback data and
> context to the completion function BTLs need to modify the
> descriptor. This could be a disaster in a multi-threaded application if
> one thread is calling the completion callback while another thread is
> preparing to start a put/get operation. To avoid issues it is better to
> provide the callback data as seperate arguments.
>
> The change is straightforward and the commit will update all BTLs and
> BTL users to use the new completion callback signature.
>
>
> When: As this was discussed at the developer's meeting last month I am
> setting a short timeout for this RFC. This times out next Wed (July
> 16).
>
>
> I would really like feedback on this change. Can it be improved? Should
> the segment data be passed back to the function (not something I need
> for RMA but might be useful elsewhere)? Would it be better to remove the
> simultaneous RDMA feature in favor of a lightweight descriptor clone (I
> have this implemented as well and I have no problem with providing
> both features)?
>
>
> This is another is a series of RFCs to improve (I hope) the BTL
> interface for one-sided operations. The next RFC will be on the
> one-sided BTL interface.
>
> -Nathan Hjelm
> HPC-5, LANL
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/07/15101.php
>
