Thanks Jason, I've just tried applying that to my local copy of HDF (using the Id constructor of DataSpace), I can't say for sure - the memory usage of the program still rose to 5GB+, but it seemed better...
If I get a chance today I'll rewrite using the C api and see what that changes. - Jorj On Fri, 14 Aug 2015 at 03:39 Jason Newton <[email protected]> wrote: > Bug found (in C++ api as usual) > > The C++ API *should* take care of inc/dec ref appropriately although they > do this in each object class (may be higher in some class hierarchies like > datatypes) but something of a leaf otherwise, rather than through > inheritance of IdComponent. That strategy while working has left a few bugs > I've found / encountered both as leaks and dec'reffing references not > incref'd. As of 1.8.15, all that I was aware of though but this concern > should be warranted all the time based on past-burnings (this would be the > third time noticing something a shared_ptr like class/wrapper around HDF > resources (IdComponent...?) would completely eliminate. > > > dataset.getSpace() leaks a reference: > > //create dataspace object using the existing id then return the object > DataSpace data_space; <--default constructor makes a valid hdf > dataspace for H5S_SCALAR > f_DataSpace_setId(&data_space, dataspace_id); <-- evil line, why didn't > we just use the ctor that takes the id parameter? > return( data_space ); > > > > //-------------------------------------------------------------------------- > // Function: f_DataSpace_setId - friend > // Purpose: This function is friend to class H5::DataSpace so that it > can > // can set DataSpace::id in order to work around a problem > // described in the JIRA issue HDFFV-7947. > // Applications shouldn't need to use it. > // param dspace - IN/OUT: DataSpace object to be changed > // param new_id - IN: New id to set > // Programmer Binh-Minh Ribler - 2015 > > //-------------------------------------------------------------------------- > void f_DataSpace_setId(DataSpace* dspace, hid_t new_id) <--evil function > that shouldn't exist (as a friend no-less!) > { > dspace->id = new_id; <-- why not dspace->p_setId(new_id);? Just make > it public already as "reset" and get rid of the friend. Follow shared_ptr > semantics.. and bring all this stuff inside IdComponent. > . > } > > > -Jason > > On Thu, Aug 13, 2015 at 9:37 AM, Miller, Mark C. <[email protected]> > wrote: > >> Hmm. Well I have no experience with HDF5's C++ interface. >> >> My first thought when reading your description was. . . I've seen that >> before. It happens when I forgot to H5Xclose() all the objects I H5Xopened >> (groups, datasets, types, dataspaces, etc.). >> >> However, with C++, I presume the interface is designed to close objects >> when they fall out of scope (e.g. deconstructor is called). So, in looking >> at your code, even though I don't see any explicit calls to close objects >> previously opened, I assume that *should* be happening when the objects >> fall out of scope. But, are you *certain* that *is* happening? Just before >> exiting main, you migth wanna make a call to H5Fget_obj_count() to get some >> idea how many objects HDF5 library thinks are still open in the file. If >> you get a large number, then that would suggest the problem is that the C++ >> interface isn't somehow closing objects as they fall out of scope. >> >> Thats all I can think of. Sorry if no help. >> >> Mark >> >> >> From: Hdf-forum <[email protected]> on behalf of Jorj >> Pimm <[email protected]> >> Reply-To: HDF Users Discussion List <[email protected]> >> Date: Thursday, August 13, 2015 9:21 AM >> To: "[email protected]" <[email protected]> >> Subject: [Hdf-forum] Growing memory usage in small HDF program >> >> Hello, >> >> I am writing an application which writes large data sets to HDF5 files, >> in fixed size blocks, using the HDF C++ API (version 1.8.15, patch 1, built >> in msvc 2013 x64) >> >> I my application seems to quickly consume all the available memory on my >> system (win32 - around 5.9GB), and then crash whenever the system becomes >> stressed (windows kills it as it has no memory) >> >> I have also tested the application on a linux machine, where I saw >> similar results. >> >> I was under the impression that by using HDF5, the file would be brought >> in and out of memory in such a way that the library would only use a small >> working set - is this not true? >> >> I have experimented with HDF features such as flushing to disk, regularly >> closing and re opening, garbage collection and tuning chunking and caching >> settings and haven't managed to get a stable working set. >> >> I've attached a minimal example, can anyone point out my mistake? >> >> Thanks, >> - Jorj >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
