On 17 Jan 2010, at 16:50, Barrett, Brian W wrote: > On Jan 17, 2010, at 11:31 AM, Ashley Pittman wrote: > >> On 10 Jan 2010, at 03:45, Barrett, Brian W wrote: >> >>> We should absolutely not change this. For simple applications, yes, things >>> work if large blocks are allocated on the heap. However, ptmalloc (and >>> most allocators, really), can't rationally cope with repeated allocations >>> and deallocations of large blocks. It would be *really bad* (as we've seen >>> before) to change the behavior of our version of ptmalloc from that which >>> is provided by Linux. Pain and suffering is all that path has ever lead to. >> >> I think there are two different issues here that are in danger of being >> confused. Having OMPI use it's own ptmalloc implantation as it has done at >> times in the past has been problematic and should be avoided in my opinion. > > I agree, and I also don't believe I'm confusing them. The shipping of > ptmalloc2 is unfortunate and yet required until a better solution appears. > The proposal/request which started this e-mail chain was to change a > hard-coded constant in the ptmalloc2 shipped with Open MPI.
I was under the impression that OMPI had moved away from shipping it's own allocator by default although I don't follow that closely. Ultimately whatever the default there needs to be a way for the end-user to change the setting. > While the fact that we can do this is an artifact of Open MPI providing it's > own ptmalloc implementation. No it isn't, it's also standard libc option. It was this that we set by default at Quadrics but the mallopt() function. > it none the less fundamentally changes how the memory allocator works when > applications are linked against Open MPI, away from the default behavior > found on Linux systems. This isn't necessarily a bad thing, default behaviour for a distro has to be something that works well for all environments that use that distro, HPC is something very specialised so there is no reason to believe the defaults are good choices for HPC apps. >> Tuning the libc malloc implementation using the options they provide to do >> is is valid and provides real benefit to a lot of applications. For the >> record we used to disable mmap based allocations by default on Quadrics >> systems and I can't think of a single case of people who needed to re-enable >> it. > > I'm glad that you've been so fortunate. Unfortunately, I have a large > application base in which that is not the case, and we have had to use mmap > based allocation, otherwise memory usage would essentially grow without > bound. So we go back to the age-old debate of correctness vs. performance. It is something bear in mind when analysing application performance and diagnosing memory problems with applications, a FAQ entry should be considered the minimum IMHO. Ashley.
