Hi all -
We finally broke down today on the telecon and decided that there's
just no way around playing memory manager tricks to get good IB and
Myrinet performance. I added two things to opal today - a dispatch
system so that different functions could register to receive
callbacks whenever the process is about to "release' memory,
containing start and length data and the ptmalloc2 memory manager.
Note that "release" is very vague - this could mean free() has been
called by the user but the process is going to hold on to the memory
or it could mean that the process is giving the memory back to the
operating system - it all depends on what the back end is capable
of. The ptmalloc2 memory manager is currently the only system we
have for intercepting release of memory, and it must be enabled
explicitly at configure time with --with-memory-manager=ptmalloc2.
There is a really simple example of using the system in topdir/test/
memory/opal_memory_basic.c.
I plan on adding a couple more backends to experiment with various
systems and their advantages / disadvantages:
- ld preload a shared object to intercept sbrk / munmap
- (possibly) a system to use the GLIBC hooks, although book
keeping might make this impractical
- Something with Darwin. Need to get back in touch with some
Apple engineers on how to do this
in a way that sucks less.
One word of caution - if you register a handler from a component, you
*must* unregister the handler before your component is closed.
Otherwise, the process is going to segfault when it tries to call the
handler after your component is unloaded.
Brian
--
Brian Barrett
Open MPI developer
http://www.open-mpi.org/
