It is MPICH, but its not a memory leak (technically speaking). Nichols A. Romero, Ph.D. Argonne Leadership Computing Facility Argonne National Laboratory Building 240 Room 2-127 9700 South Cass Avenue Argonne, IL 60490 (630) 252-3441
----- Original Message ----- From: "Quincey Koziol" <[email protected]> To: "HDF Users Discussion List" <[email protected]> Sent: Monday, August 29, 2011 10:47:00 AM Subject: Re: [Hdf-forum] memory usage in H5Dwrite On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote: > I have investigated further, its definitely an MPI issue. Thanks for taking > a look at this. Great, happy to help out. BTW, which MPI implementation has the memory leak issue? Quincey > Nichols A. Romero, Ph.D. > Argonne Leadership Computing Facility > Argonne National Laboratory > Building 240 Room 2-127 > 9700 South Cass Avenue > Argonne, IL 60490 > (630) 252-3441 > > > ----- Original Message ----- > From: "Quincey Koziol" <[email protected]> > To: "HDF Users Discussion List" <[email protected]> > Sent: Thursday, August 25, 2011 10:14:38 AM > Subject: Re: [Hdf-forum] memory usage in H5Dwrite > > Hi Nichols, > > On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote: > >> This is the test case that I sent to IBM for analysis. They assert that there >> is no memory leaked according to mallinfo. However, mmap/munmap are not >> recorded >> to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate >> the >> addresses, then I see heap memory (O(100 MB)) which is not freed. > > Yes, it doesn't look like memory should be leaked, given your usage. > Can you push the leak analysis further down inside HDF5, to see if it's in > the MPI library, or actually in HDF5? > > Thanks, > Quincey > >> Nichols A. Romero, Ph.D. >> Argonne Leadership Computing Facility >> Argonne National Laboratory >> Building 240 Room 2-127 >> 9700 South Cass Avenue >> Argonne, IL 60490 >> (630) 252-3441 >> >> >> ----- Original Message ----- >> From: "Quincey Koziol" <[email protected]> >> To: "HDF Users Discussion List" <[email protected]> >> Sent: Tuesday, August 16, 2011 2:49:05 PM >> Subject: Re: [Hdf-forum] memory usage in H5Dwrite >> >> Hi Nichols, >> >> On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote: >> >>> Hi, >>> >>> I am trying to figure out if there is a memory leak or a lack of >>> understanding on my part >>> with respect to the HDF5 internals. My simple test case writes a >>> 4-dimensional uniformly >>> distributed array using H5Dwrite. I measure the heap memory before and >>> after the write. >>> I would expect the heap memory utilization to not change substantially. >>> >>> I have measured the heap memory a couple of different ways, e.g. using >>> mallinfo or some tool. >>> Naively it looks like the MPI I/O aggregators are not releasing memory but >>> some tools >>> have reported no leak at all, while others have report a large one (> 200 >>> MB). I finally >>> resorted to using mpirun -strace to track the series of mmap/munmap in the >>> H5Dwrite. >>> I was surprised to see a large number of calls to mmap/munmap which >>> ultimately results >>> in 60 MB of heap memory not being released. >>> >>> First question: Is this the intended behavior? >>> >>> Second question: If this behavior is intentional, is there away to force >>> HDF5 to release >>> these internal buffers after the H5Dwrite. >> >> It's a bit hard to tell from your description - could you send a short >> program that demonstrates the issue? >> >> Quincey >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> <hdf5_4D.tar>_______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
