Hello Galen, Finally I've got some time to look through the new code. I have couple of notes. In pml_ob1_rdma.c you try to merge registrations in the number of places. The code looks like this: btl_mpool->mpool_deregister(btl_mpool, reg); btl_mpool->mpool_register(btl_mpool, new_base, new_len, MCA_MPOOL_FLAGS_CACHE, ®); How do you know reg is not in use? You can't deregister it if somebody is using the registration! Also I thought about merging registration and I am not sure this is such a good idea. The registration may grow to large and you will not be able to shrink it if only small part of it is in use. This may cause the waste of memory.
In mca_mpool_base_registration_t structure you save base/bound in byte granularity, but we know that kernel works in much coarse resolution. Why not to exploit this fact. We can round base/bound to page boundaries. We are going to pin this memory anyway. In my patch I introduced mpool_pageshift for this. -- Gleb.