> The problem is that MPI needs to be aware of the application doing > the free() and unregister or flush its MR cache for that virtual > address range. Of course it would be difficult for OpenMPI to have > callbacks or hooks into every way memory could be allocated/freed > that an application might use.
There are only three calls that affect the way VM memory maps to physical and thus would invalidate the mr cache: mmap, munmap and brk. Specifically what must be happening is the app registers memory, calls munmap on it, then gets the same VA back from mmap and the kernel level mr is still pointing to the original mmap: foo = mmap(...); ibv_reg_mr(mr,foo) munmap(foo..) mmap(...) == foo; // By chance due to VA randomization // Ooops, mr no longer matches proc/self/maps Actually, maybe that is the simple answer here - have the kernel fixup the mr before returning from the 2nd mmap. Then the cache in user space is still correct to assume that VA XX is registered and working. Removing entries from the registration cache would have to be done in some other way (age?). Jason _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
