Anyone care to explain?
All I want is to see how non-eager RDMA can be used by ob1 if
leave_pinned is FALSE.
2009/9/29 Roman Cheplyaka <roman.cheply...@gmail.com>:
> Thanks for your explanation, George. However:
> suppose we have leave_pinned = FALSE. Then we go to
> mca_mpool_rdma_find. There we try to find suitable registration in the
> cache. Suppose we cannot (cache's empty). Then NULL registration is
> returned, BTL is skipped and 0 is returned from mca_pml_ob1_rdma_btls.
> This way RDMA never happends.
>
> (To make it clear: this is not some theoretical situation, this is
> what I'm actually observing. OpenMPI 1.2.9) Can somebody explain
> please?
>
> 2009/9/29 George Bosilca <bosi...@eecs.utk.edu>:
>> Roman,
>>
>> Before going into explaining the logic, let me state that the memory is
>> registered (if required/supported) by the BTLs. However, this is done only
>> at the moment when the memory segment is involved in any kind of
>> communication.
>>
>> We do not want to replicate this at the PML level, in order to make sure
>> that the amount of memory registered at any moment is minimal. In other
>> words, the PML leave the decision on when to register and when to unregister
>> to the BTLs. However, in order to speedup the code a little bit (and to keep
>> things tidy), the PML will help the BTLs to work around the memory
>> registration issue. And the code you pointed out is exactly the place where
>> we do it.
>>
>> We need to register the memory if leave_pinned is TRUE, as registering will
>> leave a trace. If leave_pinned is FALSE then we only check if somehow this
>> memory is not already registered (by some BTL). In this case, there is no
>> need to create a registration in the PML (if required the BTL will do it
>> when needed).
>>
>> george.
>>
>> On Sep 28, 2009, at 13:44 , Roman Cheplyaka wrote:
>>
>>> Hi,
>>> I'm trying to dig into OpenMPI sources but have some problems. Can
>>> anyone explain the logic of the following code from
>>> mca/pml/ob1/pml_ob1_rdma.c please?
>>>
>>> if(!mca_pml_ob1.leave_pinned) {
>>> /* look through existing registrations */
>>> btl_mpool->mpool_find(btl_mpool, base, size, ®);
>>> } else {
>>> /* register the memory */
>>> btl_mpool->mpool_register(btl_mpool, base, size, 0, ®);
>>> }
>>>
>>> It seems to me that we should register new memory if leave_pinned is
>>> FALSE (i.e. no existing registrations avaliable) and search through
>>> existing registrations otherwise, but the logic is inversed here.
--
Roman I. Cheplyaka
