On Mon, 10 Dec 2007, Peter Wong wrote:
Open MPI defines its own malloc (by default), so malloc of glibc
is not called.
But, without calling malloc of glibc, the allocator of libhugetlbfs
to back text and dynamic data by large pages, e.g., 16MB pages on
POWER systems, is not used.
Indeed, we can build Open MPI with --with-memory-manager=none.
I am wondering the feasibility of turning the memory manger on
and off dynamically at runtime as a new feature?
Hi Peter -
The problem is that we actually intercept the malloc() call, so once we've
done that (which is a link-time thing), it's too late to use the
underlying malloc to actually do its thing.
I was going to add some code to Open MPI to make it an application link
time choice (rather than an OMPI-build time choice), but unfortunately
my current day to day work is not on Open MPI, so unless someone else
picks it up, it's unlikely this will get implemented in the near future.
Of course, if someone has the time and desire, I can describe to them what
I was thinking.
The only way I've found to do memory tracking at run-time is to use
LD_PRELOAD tricks, which I believe there were some other (easy to
overcome) problems with.
What would be really nice (although unlikely to occur) is if there was a
thread-safe way to hook into the memory manager directly (rather than
playing linking tricks). GLIBC's malloc provides hooks, but they aren't
thread safe (as in two user threads calling malloc at the same time would
result in badness). Darwin/Mac OS X provides thread-safe hooks that work
very well (don't require linker tricks and can be turned off at run-time),
but are slightly higher level than what we want -- there we can intercept
malloc/free, but what we'd really like to know is when memory is being
given back to the operating system.
Hope this helps,
Brian
