Sorry, “We saw the following problem in OMPI_FREE_LIST_GET_MT…”.
*From:* Владимир Трущин [mailto:vdtrusc...@compcenter.org] *Sent:* Wednesday, September 16, 2015 10:09 PM *To:* 'Open MPI Developers' *Subject:* RE: [OMPI devel] The issue with OMPI_FREE_LIST_GET_MT() George, You are right. The sequence of calls in our test is MPI_Irecv -> mca_pml_ob1_irecv -> MCA_PML_OB1_RECV_REQUEST_ALLOC. We will try to use OMPI_FREE_LIST_WAIT_MT. We saw the following problem in OMPI_FREE_LIST_WAIT_MT. It returned NULL in case when thread A was suspended after the call of ompi_free_list_grow. At this time others threads took all items from free list at the first call of opal_atomic_lifo_pop in macro. So, when thread A was unsuspended and call the second opal_atomic_lifo_pop in macro - it returned NULL. Best regards, Vladimir. *From:* devel [mailto:devel-boun...@open-mpi.org <devel-boun...@open-mpi.org>] *On Behalf Of *George Bosilca *Sent:* Wednesday, September 16, 2015 7:00 PM *To:* Open MPI Developers *Subject:* Re: [OMPI devel] The issue with OMPI_FREE_LIST_GET_MT() Alexey, This is not necessarily the fix for all cases. Most of the internal uses of the free_list can easily accommodate to the fact that no more elements are available. Based on your description of the problem I would assume you encounter this problem once the MCA_PML_OB1_RECV_REQUEST_ALLOC is called. In this particular case the problem is that fact that we call OMPI_FREE_LIST_GET_MT and that the upper level is unable to correctly deal with the case where the returned item is NULL. In this particular case the real fix is to use the blocking version of the free_list accessor (similar to the case for send) OMPI_FREE_LIST_WAIT_MT. It is also possible that I misunderstood your problem. IF the solution above doesn't work can you describe exactly where the NULL return of the OMPI_FREE_LIST_GET_MT is creating an issue? George. On Wed, Sep 16, 2015 at 9:03 AM, Алексей Рыжих <avryzh...@compcenter.org> wrote: Hi all, We experimented with MPI+OpenMP hybrid application (MPI_THREAD_MULTIPLE support level) where several threads submits a lot of MPI_Irecv() requests simultaneously and encountered an intermittent bug OMPI_ERR_TEMP_OUT_OF_RESOURCE after MCA_PML_OB1_RECV_REQUEST_ALLOC() because OMPI_FREE_LIST_GET_MT() returned NULL. Investigating this bug we found that sometimes the thread calling ompi_free_list_grow() don’t have any free items in LIFO list at exit because other threads retrieved all new items at opal_atomic_lifo_pop() So we suggest to change OMPI_FREE_LIST_GET_MT() as below: #define OMPI_FREE_LIST_GET_MT(fl, item) \ { \ item = (ompi_free_list_item_t*) opal_atomic_lifo_pop(&((fl)->super)); \ if( OPAL_UNLIKELY(NULL == item) ) { \ if(opal_using_threads()) { \ int rc; \ opal_mutex_lock(&((fl)->fl_lock)); \ do \ { \ rc = ompi_free_list_grow((fl), (fl)->fl_num_per_alloc); \ if( OPAL_UNLIKELY(rc != OMPI_SUCCESS)) break; \ \ item = (ompi_free_list_item_t*) opal_atomic_lifo_pop(&((fl)->super)); \ \ } while (!item); \ opal_mutex_unlock(&((fl)->fl_lock)); \ } else { \ ompi_free_list_grow((fl), (fl)->fl_num_per_alloc); \ item = (ompi_free_list_item_t*) opal_atomic_lifo_pop(&((fl)->super)); \ } /* opal_using_threads() */ \ } /* NULL == item */ \ } Another workaround is to increase the value of pml_ob1_free_list_inc parameter. Regards, Alexey _______________________________________________ devel mailing list de...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post: http://www.open-mpi.org/community/lists/devel/2015/09/18039.php
