On Aug 7, 2008, at 3:20 AM, Jan Ploski wrote:

    http://www.open-mpi.org/community/lists/users/2008/07/6161.php

Thanks for this explanation. According to what you wrote,
--without-memory-manager can make my and other applications run
significantly slower.

*If* they continually re-use the same buffers for large messages and you're using an OpenFabrics network (or older Myrinet/GM). Then yes, your peak bandwidth with be a bit lower. We use a pipelined protocol to offset the total impact, so it's not a total disaster -- but there is definitely some loss off the total bandwidth you'll see for large messages. Small message performance is not impacted because those use pre-registered buffers.

While I can find out just how much for my app, I
hardly can do it for other (unknown) users. So it would be nice if my heap profiling problem could be resolved in another way in the future. Is the
planned mpi_leave_pinned change in v1.3 going to correct it?

Here's what we're doing for v1.3 (it's an open ticket to document this before v1.3 is released):

1. leave_pinned will remain off by default on networks that don't care about it (tcp, sm, ...etc.)

2. for openfabrics networks (i.e., the openib btl), leave_pinned will automatically enable itself if:
    a) one of the following two is true
- you added -lopenmpi-malloc to the link step when creating your MPI app - or, if libopenmpi-malloc is not present, if a mallopt() hint [that we try by default] was able to assert itself that tells the allocator to never return memory to the OS
    b) and the user did not manually specific leave_pinned=0

libopenmpi-malloc is the same ptmalloc2 library that we used in the v1.2 series. We can detect if it is there at run-time; if it is not, we try the mallopt hint.

So by default, you shouldn't need to specify leave_pinned=1 anymore for the openib btl -- you should get the same end result as if you had specified it in the v1.2 series (better bandwidth for large messages). But if you want exactly the same behavior as v1.2, you'll need to add -lompi-malloc to your link line. We anticipate that most users won't notice the difference.

The reason we separated out the ptmalloc2 library is that it definitely causes problems with some applications (e.g., valgrind profiling), and it penalizes networks that don't need a memory allocator (e.g., tcp, sm, ...etc.).

Does that make sense?

If you'd like to give it a whirl, the trunk nightly tarballs are fairly stable at this point -- except for the sm btl. There's a bug in how the sm btl interacts with our message passing engine (ob1) that George swears he will have fixed by tomorrow. :-) But if you avoid using the sm btl, you should be fine.

    http://www.open-mpi.org/nightly/trunk/

Trunk and v1.3 snapshot tarballs are pretty close at this point; we're mainly applying bug fixes to the trunk and moving them over to the v1.3 after they've "soaked" on the trunk for a few days. Soon enough, however, the trunk will likely start diverging from the v1.3 branch as we move on towards v1.4.

--
Jeff Squyres
Cisco Systems

Reply via email to