Hi Matthew,
Try this. . .
extern herr_t H5T_init(void);
extern int H5T_term_interface(void);
int main(int argc, char *argv[])
{
.
.
.
H5Fclose(file);
H5allocate_memory(sizeof(struct H5F_t), 0);
H5T_term_interface();
H5T_init();
.
.
.
Mark
From: Hdf-forum
<[email protected]<mailto:[email protected]>>
on behalf of "Xavier, Matthew"
<[email protected]<mailto:[email protected]>>
Reply-To: HDF Users Discussion List
<[email protected]<mailto:[email protected]>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List
<[email protected]<mailto:[email protected]>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths
Yes, the H5allocate_memory() call is kind of sloppy. I could get the same
effect by putting all of those allocations in a list and freeing them before
the process exits.
As you may note, I’m not allocating that memory to store something; I’m
allocating it to prevent the allocator from giving the same block back to HDF5
on a subsequent request. It’s the same reason that I set the free list limits
to zero. If the H5F_t structure when the file is re-opened lands on the same
address, then the vlen type equality check passes because the dangling pointer
in the cached conversion path happens to match. In that case, HDF5 doesn’t
allocate a new conversion path.
So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not
see any errors.
Matthew
From: Hdf-forum [mailto:[email protected]] On Behalf Of
Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List
<[email protected]<mailto:[email protected]>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths
Hi Matthew,
Ok, I ran your code under valgrind and cannot confirm your leak report.
I was doing this against 1.8.16 HDF5 library
I *do* show a leak in your client code due to the H5allocate_memory() call with
no matching H5free_memory() call. When I add a matching H5free_memory() call,
the leak valgrind *does* report does away. . .
Valgrind run *WITH* H5free_memory()…
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak
-I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include
-L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env
LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind
--leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes
allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c
valgrind run *WITHOUT* H5free_memory().
surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak
-I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include
-L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env
LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind
--leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242
bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:
From: Hdf-forum
<[email protected]<mailto:[email protected]>>
on behalf of David <[email protected]<mailto:[email protected]>>
Reply-To: HDF Users Discussion List
<[email protected]<mailto:[email protected]>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List
<[email protected]<mailto:[email protected]>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths
Matthew,
If you're using MSVC it has some useful tools for pinpointing memory leaks. See
this article for an example:
http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I
believe there are ways to enact the dump only on changes that happened between
specific points in your code which would let you find something that is getting
cleaned up on exit but still shouldn't be happening. Sorry I can't remember the
specific calls for that but it's in the same part of the API.
David
On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew
<[email protected]<mailto:[email protected]>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t
know the exact semantics of –show-reachable), but the memory profilers I’ve
tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the
conversion path cache. I found it because I was chasing a bug report consisting
of “Why does heap memory grow by 50MB each time I open and then close a
[file]?” (A file in this case is actually a more complicated structure and may
cause hundreds of HDF5 files to be accessed.)
I think that the presence of a vlen field is important. As I recall, the type
comparison for compound types does not check what files the types were defined
in, but if a vlen field is present, then the files in which the vlen field
types were defined can make two otherwise identical compound types compare as
different.
I have not used committed datatypes. I will take a look at that function and
see if it changes the behavior.
Matthew
From: Hdf-forum
[mailto:[email protected]<mailto:[email protected]>]
On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List
<[email protected]<mailto:[email protected]>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths
I use compound types a lot *but*not* with vlen fields.
I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and
do not observe the leak you describe.
But, I also 'commit' the types to the files they are defined in.
I wonder if commiting the types would make a difference in the memory behavior
you are seeing?
Mark
From: Hdf-forum
<[email protected]<mailto:[email protected]>>
on behalf of "Xavier, Matthew"
<[email protected]<mailto:[email protected]>>
Reply-To: HDF Users Discussion List
<[email protected]<mailto:[email protected]>>
Date: Monday, March 28, 2016 2:15 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths
Some of the datasets I have contain compound types with vlen fields. When I
read these datasets, HDF5 creates new conversion ‘paths’ to convert between the
file types and memory types involved. HDF5 caches these paths (see struct H5T_g
defined in H5T.c).
I’ve finally traced a memory ‘leak’ in my application to the unbounded growth
of the conversion path cache. HDF5 treats vlen types as different if they come
from different files, so I get a new set of conversion paths for every file I
open, even if the types are actually identical.
That would be fine, except that I can’t find a way to get rid of the cached
paths when I close a file. There is a function provided for removing paths,
H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for
compound types because of the way that the pers parameter is handled. If I pass
H5T_PERS_HARD, no compound type conversions are removed because
H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the
compound->compound soft conversion and generates a new path. Alternately, if I
pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default
compound->compound soft conversion and I can't read any more datasets because
the library can't create conversion paths for them.
Incidentally, I also discovered that the way the type comparison function
determines file identity depends on pointers that are left dangling when a file
is closed, which both complicated my minimum reproduction of the problem and
also undermines the file identity check. (When the same allocation is re-used
from the free list for a different file, types can compare as the same even
when they are from different files, which is contrary to the intent of the
code.)
I have attached a small program that reproduces the problem. It takes one
argument, which is a path at which it can write a temporary file. To run it
does require a custom build of HDF5 so that the test program can read the size
of the path table. (Or alternately, you can comment out the relevant parts of
the test program and inspect H5T_g.npaths with a debugger.)
Has anyone else encountered and/or found a solution for this problem? I am
already patching my own HDF5 builds to get Unicode file name support on
Windows, so if I have to make code changes it’s not the end of the world.
Thanks,
Matthew Xavier
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]<mailto:[email protected]>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
