Not really... there are two things which are causing problems. 1. glibc malloc on BG/P does not release memory if it goes into the arena, the threshold between mmap and arena is different on BG/P that on Linux
2. 4D array is being sliced into 4D chunks which leads to a large offsets in the ROMIO driver (this is somehow problematic for memory, don't know the details here, someone at ANL from the MPICH will look into this). Nichols A. Romero, Ph.D. Argonne Leadership Computing Facility Argonne National Laboratory Building 240 Room 2-127 9700 South Cass Avenue Argonne, IL 60490 (630) 252-3441 ----- Original Message ----- From: "Quincey Koziol" <[email protected]> To: "HDF Users Discussion List" <[email protected]> Sent: Monday, August 29, 2011 3:11:55 PM Subject: Re: [Hdf-forum] memory usage in H5Dwrite On Aug 29, 2011, at 3:02 PM, Nichols A. Romero wrote: > It is MPICH, but its not a memory leak (technically speaking). Ah, something in the library that is only freed at MPI_Finalize? Quincey > Nichols A. Romero, Ph.D. > Argonne Leadership Computing Facility > Argonne National Laboratory > Building 240 Room 2-127 > 9700 South Cass Avenue > Argonne, IL 60490 > (630) 252-3441 > > > ----- Original Message ----- > From: "Quincey Koziol" <[email protected]> > To: "HDF Users Discussion List" <[email protected]> > Sent: Monday, August 29, 2011 10:47:00 AM > Subject: Re: [Hdf-forum] memory usage in H5Dwrite > > > On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote: > >> I have investigated further, its definitely an MPI issue. Thanks for taking >> a look at this. > > Great, happy to help out. BTW, which MPI implementation has the memory > leak issue? > > Quincey > >> Nichols A. Romero, Ph.D. >> Argonne Leadership Computing Facility >> Argonne National Laboratory >> Building 240 Room 2-127 >> 9700 South Cass Avenue >> Argonne, IL 60490 >> (630) 252-3441 >> >> >> ----- Original Message ----- >> From: "Quincey Koziol" <[email protected]> >> To: "HDF Users Discussion List" <[email protected]> >> Sent: Thursday, August 25, 2011 10:14:38 AM >> Subject: Re: [Hdf-forum] memory usage in H5Dwrite >> >> Hi Nichols, >> >> On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote: >> >>> This is the test case that I sent to IBM for analysis. They assert that >>> there >>> is no memory leaked according to mallinfo. However, mmap/munmap are not >>> recorded >>> to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate >>> the >>> addresses, then I see heap memory (O(100 MB)) which is not freed. >> >> Yes, it doesn't look like memory should be leaked, given your usage. >> Can you push the leak analysis further down inside HDF5, to see if it's in >> the MPI library, or actually in HDF5? >> >> Thanks, >> Quincey >> >>> Nichols A. Romero, Ph.D. >>> Argonne Leadership Computing Facility >>> Argonne National Laboratory >>> Building 240 Room 2-127 >>> 9700 South Cass Avenue >>> Argonne, IL 60490 >>> (630) 252-3441 >>> >>> >>> ----- Original Message ----- >>> From: "Quincey Koziol" <[email protected]> >>> To: "HDF Users Discussion List" <[email protected]> >>> Sent: Tuesday, August 16, 2011 2:49:05 PM >>> Subject: Re: [Hdf-forum] memory usage in H5Dwrite >>> >>> Hi Nichols, >>> >>> On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote: >>> >>>> Hi, >>>> >>>> I am trying to figure out if there is a memory leak or a lack of >>>> understanding on my part >>>> with respect to the HDF5 internals. My simple test case writes a >>>> 4-dimensional uniformly >>>> distributed array using H5Dwrite. I measure the heap memory before and >>>> after the write. >>>> I would expect the heap memory utilization to not change substantially. >>>> >>>> I have measured the heap memory a couple of different ways, e.g. using >>>> mallinfo or some tool. >>>> Naively it looks like the MPI I/O aggregators are not releasing memory but >>>> some tools >>>> have reported no leak at all, while others have report a large one (> 200 >>>> MB). I finally >>>> resorted to using mpirun -strace to track the series of mmap/munmap in the >>>> H5Dwrite. >>>> I was surprised to see a large number of calls to mmap/munmap which >>>> ultimately results >>>> in 60 MB of heap memory not being released. >>>> >>>> First question: Is this the intended behavior? >>>> >>>> Second question: If this behavior is intentional, is there away to force >>>> HDF5 to release >>>> these internal buffers after the H5Dwrite. >>> >>> It's a bit hard to tell from your description - could you send a short >>> program that demonstrates the issue? >>> >>> Quincey >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >>> <hdf5_4D.tar>_______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
