Eugen, Thank you for checking! I really appreciate your help. The JIRA issue is HDFFV-8645.
Elena ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elena Pourmal The HDF Group http://hdfgroup.org 1800 So. Oak St., Suite 203, Champaign IL 61820 217.531.6112 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On Dec 17, 2013, at 12:55 AM, "Wintersberger, Eugen" <[email protected]> wrote: > Hi Elena > On Mon, 2013-12-16 at 22:56 -0600, Elena Pourmal wrote: >> Hi Eugen, >> >> It may be a feature (or a bug ;-) >> >> In preparation for I/O HDF5 allocates internal structures for handling >> chunks. The overhead for each chunk is a couple Ks (cmp. with the size of >> the chunk!). After I/O is done, memory required for handling the structures >> is put on a "free" list for reuse. 3GBs seems little-bit too much and we >> will need to investigate. > > Ok. That's what I have expected. > >> >> Meanwhile, could you please try to call H5garbage_collect >> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-GarbageCollect after >> H5Dwrite to see if memory is released? > > Did not have any effect. > >> >> You may also try to write just one chunk after the "big" write. It should >> also release memory. > > No. It does not ;) - at least not in my case. > >> >> But as I said, we will need to look more closely at memory consumption and >> see if any improvements/tuning/fixes could be done. I'll add the issue to >> our database. > > Attached is a new version of the program I have used for testing. > > regards > Eugen > >> >> Thank you! >> >> Elena >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Elena Pourmal The HDF Group http://hdfgroup.org >> 1800 So. Oak St., Suite 203, Champaign IL 61820 >> 217.531.6112 >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> >> >> >> On Dec 11, 2013, at 2:33 AM, "Wintersberger, Eugen" >> <[email protected]> wrote: >> >>> Hi there >>> we (a colleague and I) stumbled upon an interesting behavior of the >>> HDF5 library (we are using 1.8.11 on Debian Wheezy). >>> We try to write a single 2D array of data to an HDF5 dataset with an >>> absolutely stupid chunking scheme (see the test() function in the >>> attached source file). As a result the library allocates quite a lot of >>> memory (around 3 GByte). What surprised us is that this memory is not >>> freed even after closing the file. Moreover, it does not grow when >>> calling the test() several times as can be seen in the output of the >>> attached program >>> >>> ./test >>> Startup ... >>> RSS - 6.880000e+02 kB >>> Shared Memory - 5.200000e+02 kB >>> Private Memory - 1.680000e+02 kB >>> >>> After first write ... >>> RSS - 2.916884e+06 kB >>> Shared Memory - 2.160000e+03 kB >>> Private Memory - 2.914724e+06 kB >>> >>> After second write ... >>> RSS - 2.921896e+06 kB >>> Shared Memory - 2.160000e+03 kB >>> Private Memory - 2.919736e+06 kB >>> >>> Obviously this is not a resource leak in the classical sense. My >>> suspicion is that the memory is occupied by some persistent cache. >>> Which leads me to my question: is there a possibility to free this >>> memory? >>> >>> regards >>> Eugen >>> >>> -- >>> --------------------------------------- >>> DI. Dr. Eugen Wintersberger >>> >>> FS-EC >>> DESY >>> Notkestr. 85 >>> D-22607 Hamburg >>> Germany >>> >>> E-Mail: [email protected] >>> Telefon: +49-40-8998-1917 >>> --------------------------------------- >>> <test.c>_______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > <test.c>_______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
