On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote:
> I have investigated further, its definitely an MPI issue. Thanks for taking
> a look at this.
Great, happy to help out. BTW, which MPI implementation has the memory
leak issue?
Quincey
> Nichols A. Romero, Ph.D.
> Argonne Leadership Computing Facility
> Argonne National Laboratory
> Building 240 Room 2-127
> 9700 South Cass Avenue
> Argonne, IL 60490
> (630) 252-3441
>
>
> ----- Original Message -----
> From: "Quincey Koziol" <[email protected]>
> To: "HDF Users Discussion List" <[email protected]>
> Sent: Thursday, August 25, 2011 10:14:38 AM
> Subject: Re: [Hdf-forum] memory usage in H5Dwrite
>
> Hi Nichols,
>
> On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:
>
>> This is the test case that I sent to IBM for analysis. They assert that there
>> is no memory leaked according to mallinfo. However, mmap/munmap are not
>> recorded
>> to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate
>> the
>> addresses, then I see heap memory (O(100 MB)) which is not freed.
>
> Yes, it doesn't look like memory should be leaked, given your usage.
> Can you push the leak analysis further down inside HDF5, to see if it's in
> the MPI library, or actually in HDF5?
>
> Thanks,
> Quincey
>
>> Nichols A. Romero, Ph.D.
>> Argonne Leadership Computing Facility
>> Argonne National Laboratory
>> Building 240 Room 2-127
>> 9700 South Cass Avenue
>> Argonne, IL 60490
>> (630) 252-3441
>>
>>
>> ----- Original Message -----
>> From: "Quincey Koziol" <[email protected]>
>> To: "HDF Users Discussion List" <[email protected]>
>> Sent: Tuesday, August 16, 2011 2:49:05 PM
>> Subject: Re: [Hdf-forum] memory usage in H5Dwrite
>>
>> Hi Nichols,
>>
>> On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:
>>
>>> Hi,
>>>
>>> I am trying to figure out if there is a memory leak or a lack of
>>> understanding on my part
>>> with respect to the HDF5 internals. My simple test case writes a
>>> 4-dimensional uniformly
>>> distributed array using H5Dwrite. I measure the heap memory before and
>>> after the write.
>>> I would expect the heap memory utilization to not change substantially.
>>>
>>> I have measured the heap memory a couple of different ways, e.g. using
>>> mallinfo or some tool.
>>> Naively it looks like the MPI I/O aggregators are not releasing memory but
>>> some tools
>>> have reported no leak at all, while others have report a large one (> 200
>>> MB). I finally
>>> resorted to using mpirun -strace to track the series of mmap/munmap in the
>>> H5Dwrite.
>>> I was surprised to see a large number of calls to mmap/munmap which
>>> ultimately results
>>> in 60 MB of heap memory not being released.
>>>
>>> First question: Is this the intended behavior?
>>>
>>> Second question: If this behavior is intentional, is there away to force
>>> HDF5 to release
>>> these internal buffers after the H5Dwrite.
>>
>> It's a bit hard to tell from your description - could you send a short
>> program that demonstrates the issue?
>>
>> Quincey
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>> <hdf5_4D.tar>_______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
