On 4/29/09 15:55 , "Jason Gunthorpe" <[email protected]> wrote:
>> The problem is that MPI needs to be aware of the application doing >> the free() and unregister or flush its MR cache for that virtual >> address range. Of course it would be difficult for OpenMPI to have >> callbacks or hooks into every way memory could be allocated/freed >> that an application might use. > > There are only three calls that affect the way VM memory maps to > physical and thus would invalidate the mr cache: mmap, munmap and brk. There's also System V shared memory, which at least one scientific code out there uses. > Specifically what must be happening is the app registers memory, calls > munmap on it, then gets the same VA back from mmap and the kernel > level mr is still pointing to the original mmap: > > foo = mmap(...); > ibv_reg_mr(mr,foo) > munmap(foo..) > mmap(...) == foo; // By chance due to VA randomization > // Ooops, mr no longer matches proc/self/maps > > Actually, maybe that is the simple answer here - have the kernel fixup > the mr before returning from the 2nd mmap. Then the cache in user > space is still correct to assume that VA XX is registered and working. Yeah, although that could get really nasty as there's generally not one call to ibv_reg_mr per call to mmap. It's usually a couple of calls to ibv_reg_mr for different segments of the same mmap buffer (think sending faces of a 3-d block of space to the nearest neighbors in a physics simulation). > Removing entries from the registration cache would have to be done in > some other way (age?). Brian -- Brian W. Barrett Dept. 1423: Scalable System Software Sandia National Laboratories _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
