[OMPI devel] OMPI_FREE_LIST improvements

[Galen Shipman]

In working on my changes in the ib_multifrag branch I modified the  
ompi_free_list.
The change enables a free list to have a bit more personality than  
what is dictated by the type of the item on the free list. The  
overall problem was that we often use different free list item types  
to simply distinguish sizes of the free list item. In an ideal world  
we would just have constructors that accepted arguments. There are  
numerous problems with this approach, but mostly it would require a  
major change to the object system, don't think we want that. So  
instead I modified the free list to allow for an optional "post  
constructor" initialization function to be run on each free list item  
which includes optional opaque data to be passed to the  
initialization routine.

Here is the signature of the initialization routine:

typedef void (*ompi_free_list_item_init_fn_t) (struct  
ompi_free_list_item_t*, void* ctx);


I also added two new items to the free list struct:

struct ompi_free_list_t
{
    .............
    ompi_free_list_item_init_fn_t item_init;
    void* ctx;
};




The current ompi_free_list_init function didn't change at all,  
instead I added these optional params to ompi_free_list_init_ex:

OMPI_DECLSPEC int ompi_free_list_init_ex(
    ompi_free_list_t *free_list,
    size_t element_size,
    size_t alignment,
    opal_class_t* element_class,
    int num_elements_to_alloc,
    int max_elements_to_alloc,
    int num_elements_per_alloc,
    struct mca_mpool_base_module_t*,
    ompi_free_list_item_init_fn_t item_init,
    void *ctx
    );


So all the free list does is run the function specified by  
"item_init" on created free list item (after calling  
OBJ_CONSTRUCT_INTERNAL)


For those that don't need this new functionality, simply pass two  
NULL's to ompi_free_lint_init_ex :


    ompi_free_list_init_ex(&btl->udapl_frag_eager,
        sizeof(mca_btl_udapl_frag_eager_t) +
            mca_btl_udapl_component.udapl_eager_frag_size,
        mca_btl_udapl_component.udapl_buffer_alignment,
        OBJ_CLASS(mca_btl_udapl_frag_eager_t),
        mca_btl_udapl_component.udapl_free_list_num,
        mca_btl_udapl_component.udapl_free_list_max,
        mca_btl_udapl_component.udapl_free_list_inc,
                           btl->super.btl_mpool,
                           NULL,
                           NULL);


Again, if you are using ompi_free_list_init you won't be affected.


I think this functionality makes sense, it reduced the number of  
different free list item types in the OpenIB BTL and allows me to  
have numerous free lists of the same item type but with slightly  
different characteristics.

Here is an example of how I use this in the OpenIB BTL:

init_data = (mca_btl_openib_frag_init_data_t*)
            malloc(sizeof(mca_btl_openib_frag_init_data_t));

  init_data->length = length;
  init_data->type = MCA_BTL_OPENIB_FRAG_SEND_USER;
  init_data->order = mca_btl_openib_component.rdma_qp;
  init_data->list = &openib_btl->send_user_free;

ompi_free_list_init_ex( &openib_btl->send_user_free,
                                           length,
                                           2,
                                           OBJ_CLASS 
(mca_btl_openib_send_user_frag_t),

mca_btl_openib_component.ib_free_list_num,

mca_btl_openib_component.ib_free_list_max,

mca_btl_openib_component.ib_free_list_inc,
                                           NULL,
                                           mca_btl_openib_frag_init,
                                           (void*)init_data))




Thanks,

Galen