On Aug 29, 2011, at 3:02 PM, Nichols A. Romero wrote:
> It is MPICH, but its not a memory leak (technically speaking).
Ah, something in the library that is only freed at MPI_Finalize?
Quincey
> Nichols A. Romero, Ph.D.
> Argonne Leadership Computing Facility
> Argonne National Laboratory
> Building 240 Room 2-127
> 9700 South Cass Avenue
> Argonne, IL 60490
> (630) 252-3441
>
>
> ----- Original Message -----
> From: "Quincey Koziol" <[email protected]>
> To: "HDF Users Discussion List" <[email protected]>
> Sent: Monday, August 29, 2011 10:47:00 AM
> Subject: Re: [Hdf-forum] memory usage in H5Dwrite
>
>
> On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote:
>
>> I have investigated further, its definitely an MPI issue. Thanks for taking
>> a look at this.
>
> Great, happy to help out. BTW, which MPI implementation has the memory
> leak issue?
>
> Quincey
>
>> Nichols A. Romero, Ph.D.
>> Argonne Leadership Computing Facility
>> Argonne National Laboratory
>> Building 240 Room 2-127
>> 9700 South Cass Avenue
>> Argonne, IL 60490
>> (630) 252-3441
>>
>>
>> ----- Original Message -----
>> From: "Quincey Koziol" <[email protected]>
>> To: "HDF Users Discussion List" <[email protected]>
>> Sent: Thursday, August 25, 2011 10:14:38 AM
>> Subject: Re: [Hdf-forum] memory usage in H5Dwrite
>>
>> Hi Nichols,
>>
>> On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:
>>
>>> This is the test case that I sent to IBM for analysis. They assert that
>>> there
>>> is no memory leaked according to mallinfo. However, mmap/munmap are not
>>> recorded
>>> to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate
>>> the
>>> addresses, then I see heap memory (O(100 MB)) which is not freed.
>>
>> Yes, it doesn't look like memory should be leaked, given your usage.
>> Can you push the leak analysis further down inside HDF5, to see if it's in
>> the MPI library, or actually in HDF5?
>>
>> Thanks,
>> Quincey
>>
>>> Nichols A. Romero, Ph.D.
>>> Argonne Leadership Computing Facility
>>> Argonne National Laboratory
>>> Building 240 Room 2-127
>>> 9700 South Cass Avenue
>>> Argonne, IL 60490
>>> (630) 252-3441
>>>
>>>
>>> ----- Original Message -----
>>> From: "Quincey Koziol" <[email protected]>
>>> To: "HDF Users Discussion List" <[email protected]>
>>> Sent: Tuesday, August 16, 2011 2:49:05 PM
>>> Subject: Re: [Hdf-forum] memory usage in H5Dwrite
>>>
>>> Hi Nichols,
>>>
>>> On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to figure out if there is a memory leak or a lack of
>>>> understanding on my part
>>>> with respect to the HDF5 internals. My simple test case writes a
>>>> 4-dimensional uniformly
>>>> distributed array using H5Dwrite. I measure the heap memory before and
>>>> after the write.
>>>> I would expect the heap memory utilization to not change substantially.
>>>>
>>>> I have measured the heap memory a couple of different ways, e.g. using
>>>> mallinfo or some tool.
>>>> Naively it looks like the MPI I/O aggregators are not releasing memory but
>>>> some tools
>>>> have reported no leak at all, while others have report a large one (> 200
>>>> MB). I finally
>>>> resorted to using mpirun -strace to track the series of mmap/munmap in the
>>>> H5Dwrite.
>>>> I was surprised to see a large number of calls to mmap/munmap which
>>>> ultimately results
>>>> in 60 MB of heap memory not being released.
>>>>
>>>> First question: Is this the intended behavior?
>>>>
>>>> Second question: If this behavior is intentional, is there away to force
>>>> HDF5 to release
>>>> these internal buffers after the H5Dwrite.
>>>
>>> It's a bit hard to tell from your description - could you send a short
>>> program that demonstrates the issue?
>>>
>>> Quincey
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>> <hdf5_4D.tar>_______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
