Thanks, Joel! I have your files and will take a look. -Barbara -----Original Message----- From: Hdf-forum [mailto:[email protected]] On Behalf Of Joel Gales Sent: Friday, February 28, 2014 2:27 PM To: HDF Users Discussion List Cc: Bryan Franz; sean bailey Subject: Re: [Hdf-forum] memory leak reading compressed files
Barbara, I've place a datafile A2003061013500.L2_LAC_AT110 on the ftp server as well as a routine compress_memleak_multifiles.c that I hope will demonstrate the problem. It is actually a netcdf4 file but the HDF5 library is called to do the reads and the example program uses only HDF5 calls. It simulates multiple files by opening the same one five times. The allocated memory leak is about 15M. We are using hdf5-1.8.10-patch1. Joel On Fri, 28 Feb 2014, Barbara Jones wrote: > > Hi Joel, > > Which version of HDF5 are you using? Would you be able to send me a small > program or code snippet and a sample file to reproduce the issue? A similar > issue came up a while back (pre-HDF5-1.8.5) with a netCDF-4 user, but we > never determined what the issue was. > > You can place the files on our local ftp server. Let me know (at > [email protected]) when they are there. > > Here are the instructions: > --------------- > ftp ftp.hdfgroup.uiuc.edu > > Login as "anonymous" > > Go to the pub/incoming/help/ directory: > cd pub/incoming/help > > Type 'pwd' to see if you are in that location. (You will not be able > to see the contents of the directory.): > pwd > > Type 'bin' to put in binary mode: > bin > > Place file in directory after you are in binary mode: > put xxx > > Get out of ftp: > quit > -------------- > > Thanks! > -Barbara > > ================= > Barbara Jones > The HDF Helpdesk > > The HDF Group > [email protected] > ================= > > > -----Original Message----- > From: Hdf-forum [mailto:[email protected]] On > Behalf Of Joel Gales > Sent: Sunday, February 23, 2014 2:01 PM > To: [email protected] > Cc: Bryan Franz; sean bailey > Subject: [Hdf-forum] memory leak reading compressed files > > I am reading a large number (186) of HDF5 compressed datafiles. > Eventually the allocated memory exceeds 20g and the program fails. > > I've traced the increasing memory allocation to the H5Z_filter_deflate() > function in H5Zdeflate.c, specifically the lines: > > nalloc *= 2; > if(NULL == (new_outbuf = H5MM_realloc(outbuf, nalloc))) { > (void)inflateEnd(&z_strm); > HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, 0, "memory > allocation failed for deflate uncompression") > > It appears as if this memory is never freed causing the memory leak. > > Is this the desired behaviour or could there be a bug that prevents this > memory from being freed? > > > ========================= > Joel Gales > Futuretech Corp. > > SIMBIOS Code 616.0 > Phone: (301) 286-1403 > FAX: (301) 286-0268 > > Bin Globally Map Locally > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgro > up.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgro > up.org > ========================= Joel Gales Futuretech Corp. SIMBIOS Code 616.0 Phone: (301) 286-1403 FAX: (301) 286-0268 Bin Globally Map Locally _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
