Re: [O-MPI devel] Registration Cache changes

[Galen Mark Shipman]

We changed things a bit in the mpool, the deregister will remove the
registration from the cache and then call release, if the reference count
is <=0, the memory is then deregistered, otherwise it will be deregistered
later via another release call. The BTL module increments the reference
count on the registration so it should not be deregestered until the the
registration's reference count is decremented in btl_free.

Are you seeing an actual vapi error with the attached code? If so it is
probably a reference count issue that we need to deal with.

Thanks,

Galen

> Hello Galen,
>
> It seams this issue is still present and can be easily triggered.
> (see attached program). Do you have plans to fix it?
>
> On Wed, Sep 21, 2005 at 12:06:18PM -0600, Galen M. Shipman wrote:
>> Gleb,
>>
>> >
>> > Gleb Natapov wrote:
>> >
>> >> Hello Galen,
>> >>
>> >> Finally I've got some time to look through the new code.
>> >> I have couple of notes.  In pml_ob1_rdma.c you try to merge
>> >> registrations in the number of places. The code looks like this:
>> >>   btl_mpool->mpool_deregister(btl_mpool, reg);
>> >>   btl_mpool->mpool_register(btl_mpool,
>> >> new_base,
>> >> new_len,
>> >> MCA_MPOOL_FLAGS_CACHE,
>> >> );
>> >> How do you know reg is not in use? You can't deregister it if
>> >> somebody
>> >> is using the registration!
>> >>
>> >
>> > Good catch... this should check the reference count and
>> > only deregister when the reference count actually goes to zero...
>> > ___
>> > devel mailing list
>> > de...@open-mpi.org
>> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> >
>>
>> Yes, this was a good catch.. This was causing all sorts of fun for us!
>> Thanks,
>>
>
> --
>   Gleb.
>

Re: [O-MPI devel] Registration Cache changes

[Tim S. Woodall]

Gleb Natapov wrote:

Hello Galen,

Finally I've got some time to look through the new code.
I have couple of notes.  In pml_ob1_rdma.c you try to merge 
registrations in the number of places. The code looks like this:

  btl_mpool->mpool_deregister(btl_mpool, reg);
  btl_mpool->mpool_register(btl_mpool, 
new_base,

new_len,
MCA_MPOOL_FLAGS_CACHE,
);
How do you know reg is not in use? You can't deregister it if somebody
is using the registration!


Good catch... this should check the reference count and
only deregister when the reference count actually goes to zero...

[O-MPI devel] Registration Cache changes

[Galen M. Shipman]

Hi Gleb,

Tim and I have incorporated some of the changes you mentioned into a  
new rcache framework. Currently there is a single component in this  
framework, rcache_rb which is a registration cache based on a red- 
black tree  and uses an MRU list if the registration is not  
persistent. Note that each mpool now has it's own rcache instead of  
the btl's sharing an rcache. This has significantly simplified both  
the btls and the pml. Thanks for your input and please feel free to  
critique.


Galen