John, I couldn't compile your program on Mac or Linux due to the usage of (/..,.../) in the function calls.
I also don't see anything obviously wrong. Could you please comment out h5pset_deflate_f call, rerun the program and check if you still see memory growth? Thank you! Elena On Jul 29, 2010, at 4:01 PM, Berkery, John (GE Energy, Non-GE) wrote: > I've got a problem with excessive memory leakage using HDF5 in a Fortran > program. Here's a simple example of adding a large fixed size matrix to > a H5 file. It Iterates through a loop where the dataspace is enlarged, > another matrix is added to the file and the dataspace is closed. > Shouldn't it free all the memory at the end of each loop and return to a > certain base memory footprint? Instead it continues to grow throughout > the process. > > Jack Berkery > GE Energy Applications > Schenectady, NY > > > > ! > ************************************************************************ > ****** > ! * > ! * Project: memtest.f90 > ! * Subject: Test appending a matrix to H5 file. > ! * > ! > ************************************************************************ > ****** > > program main > > use HDF5 > > implicit none > > integer :: i, j, h5err > integer :: nrows = 1000 > integer :: ncols = 500 > real, dimension(500,1000) :: matrix > > integer(HID_T) :: file ! File id > integer(HID_T) :: hdata ! Group id for storing hourly data > integer(HID_T) :: dset ! Dataset id > integer(HID_T) :: dspc ! Dataspace id > integer(HID_T) :: dprp ! Dataproperties id > integer(HID_T) :: dtyp ! Datatype id > integer(HID_T) :: mset ! Memory Dataset id > integer(HID_T) :: mspc ! Memory Dataspace id > > ! Variables for fetching dimension sizes from dataspaces within the > file. > integer(HSIZE_T) :: dimd(2), maxdim(2) > > ! Initialize FORTRAN interface. > call H5open_f ( h5err ) > ! Create a new file using default properties. > call H5Fcreate_f ( "test03.h5", H5F_ACC_TRUNC_F, file, h5err ) > ! Hourly data - Stores info about the settings at a given year. > call H5Gcreate_f ( file, "HData", hdata, h5err ) > ! Create dataspace to store expandable matrix of values > call H5Screate_simple_f ( 2, (/0, nrows/), dspc, h5err, > (/H5S_UNLIMITED_F, nrows/) ) > ! Define the datatype for this set > call H5Tcopy_f ( H5T_NATIVE_REAL, dtyp, h5err ) > ! Create property objects > call H5Pcreate_f ( H5P_DATASET_CREATE_F, dprp, h5err ) > ! Set properties - chunked (one at a time) > call H5Pset_chunk_f ( dprp, 2, (/1, nrows/), h5err ) > ! Set properties - using compression level 7 [0-9] (9 most > compression) > call H5Pset_deflate_f ( dprp, 7, h5err ) > ! Create the dataset with modified properties. > call H5Dcreate_f ( hdata, "vals", dtyp, dspc, dset, h5err, dprp ) > ! Create memory dataspaces to enable chunked write > call H5Screate_simple_f ( 2, (/ncols, nrows/), mspc, h5err ) > > ! Fill in data variables > do i = 1, ncols > do j = 1, nrows > matrix (i,j) = 100.0 * real(i) + real(j) > end do > end do > > ! Write dataspace in a loop > ! New data has to be appended to the dataset already in the file. > > do i = 1, 30 > print *, "Loop", i > ! Get a new pointer to a copy of the dataspace for this set > call H5Dget_space_f ( dset, dspc, h5err ) > ! Figure out how big is the dataset in the file. > call H5Sget_simple_extent_dims_f ( dspc, dimd, maxdim, h5err ) > ! Extend size of the dataset in the file > call H5Dextend_f ( dset, dimd+(/ncols,0/), h5err ) > ! Close the dataspace now that it has been extended > call H5Sclose_f ( dspc, h5err ) > ! Fetch the dataspace of the extended dataset > call H5Dget_space_f ( dset, dspc, h5err ) > ! Select the appropriate slab in the extended dataset > call H5Sselect_hyperslab_f ( dspc, H5S_SELECT_SET_F, & > (/dimd(1),0/), (/1,1/), h5err, (/1,1/), shape(matrix) ) > ! Write the data from memory to the selected slab > call H5Dwrite_f ( dset, H5T_NATIVE_REAL, matrix, shape(matrix), > h5err, & > mspc, dspc, H5P_DEFAULT_F ) > ! Close the extended dataspace to free memory > call H5Sclose_f ( dspc, h5err ) > call sleep (3) ! for a better view of process explorer > graph > end do > > ! Cleanup, end access and release resources > > call sleep (5) ! for a better view of process > explorer graph > call H5Dclose_f ( dset, h5err ) ! dataset > !call H5Sclose_f ( dspc, h5err ) ! dataspace (released inside loop) > call H5Pclose_f ( dprp, h5err ) ! properties > call H5Tclose_f ( dtyp, h5err ) ! data types > > call H5Fclose_f ( file, h5err ) ! file id > call H5close_f ( h5err ) ! fortran interface > call sleep (5) > > end program main > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
