I know. That's my problem. I'm a contractor for GE and have no authority to change any versions of libraries or compilers. I was originally working with 1.6.5 then built 1.6.10 and saw no improvement. I attempted to build 1.8.5 from source but the visual studio and fortran compilers were not up to it. Are there binary distributions of fortran libs/dlls for windows?
Jack Berkery -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Elena Pourmal Sent: Monday, August 02, 2010 11:17 AM To: HDF Users Discussion List Subject: Re: [Hdf-forum] HDF5 1.6.10 memory growth John Thank you for your report. Could you please try with the latest 1.8.5-snap4 ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/hdf5/snapshots/v18/ and see if the problem is still there? (BTW, we stopped supporting HDF5 1.6.*) I will enter a bug report and we will investigate. Thank you! Elena On Aug 2, 2010, at 7:39 AM, Berkery, John (GE Energy, Non-GE) wrote: > We do compile the main app on Linux with such imbedded data statements > but I don't know which compiler is used. I'm using visual fortran on > Windows. > > Yes, the memory allocation continues to grow without deflate but not > as much. Whereas it grew from 5.7 to 12.1 Mb before, it now grows from > 5.7 to 9.9 Mb. Compression is a must in the real app though, otherwise > the > H5 file would grow to an enormous size. There is very little > difference in memory allocation between compression level 1 and 7. > > Attached are graphs of the program runs with and without. There is a > difference in how memory is accumulating and it is inconsistent > between iterations of the loop which is also disconcerting. One would > expect a regular stair step pattern more similar to the run with compression. > > Jack Berkery > GE Energy Applications > Schenectady, NY > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Elena Pourmal > Sent: Sunday, August 01, 2010 7:14 PM > To: HDF Users Discussion List > Subject: Re: [Hdf-forum] HDF5 1.6.10 memory growth > > John, > > I couldn't compile your program on Mac or Linux due to the usage of > (/..,.../) in the function calls. > > I also don't see anything obviously wrong. Could you please comment > out h5pset_deflate_f call, rerun the program and check if you still > see memory growth? > > Thank you! > > Elena > On Jul 29, 2010, at 4:01 PM, Berkery, John (GE Energy, Non-GE) wrote: > >> I've got a problem with excessive memory leakage using HDF5 in a >> Fortran program. Here's a simple example of adding a large fixed size >> matrix to a H5 file. It Iterates through a loop where the dataspace >> is > >> enlarged, another matrix is added to the file and the dataspace is > closed. >> Shouldn't it free all the memory at the end of each loop and return >> to > >> a certain base memory footprint? Instead it continues to grow >> throughout the process. >> >> Jack Berkery >> GE Energy Applications >> Schenectady, NY >> >> >> >> ! >> ********************************************************************* >> * >> ** >> ****** >> ! * >> ! * Project: memtest.f90 >> ! * Subject: Test appending a matrix to H5 file. >> ! * >> ! >> ********************************************************************* >> * >> ** >> ****** >> >> program main >> >> use HDF5 >> >> implicit none >> >> integer :: i, j, h5err >> integer :: nrows = 1000 >> integer :: ncols = 500 >> real, dimension(500,1000) :: matrix >> >> integer(HID_T) :: file ! File id >> integer(HID_T) :: hdata ! Group id for storing hourly data >> integer(HID_T) :: dset ! Dataset id >> integer(HID_T) :: dspc ! Dataspace id >> integer(HID_T) :: dprp ! Dataproperties id >> integer(HID_T) :: dtyp ! Datatype id >> integer(HID_T) :: mset ! Memory Dataset id >> integer(HID_T) :: mspc ! Memory Dataspace id >> >> ! Variables for fetching dimension sizes from dataspaces within the > >> file. >> integer(HSIZE_T) :: dimd(2), maxdim(2) >> >> ! Initialize FORTRAN interface. >> call H5open_f ( h5err ) >> ! Create a new file using default properties. >> call H5Fcreate_f ( "test03.h5", H5F_ACC_TRUNC_F, file, h5err ) >> ! Hourly data - Stores info about the settings at a given year. >> call H5Gcreate_f ( file, "HData", hdata, h5err ) >> ! Create dataspace to store expandable matrix of values >> call H5Screate_simple_f ( 2, (/0, nrows/), dspc, h5err, >> (/H5S_UNLIMITED_F, nrows/) ) >> ! Define the datatype for this set >> call H5Tcopy_f ( H5T_NATIVE_REAL, dtyp, h5err ) >> ! Create property objects >> call H5Pcreate_f ( H5P_DATASET_CREATE_F, dprp, h5err ) >> ! Set properties - chunked (one at a time) >> call H5Pset_chunk_f ( dprp, 2, (/1, nrows/), h5err ) >> ! Set properties - using compression level 7 [0-9] (9 most >> compression) >> call H5Pset_deflate_f ( dprp, 7, h5err ) >> ! Create the dataset with modified properties. >> call H5Dcreate_f ( hdata, "vals", dtyp, dspc, dset, h5err, dprp ) >> ! Create memory dataspaces to enable chunked write >> call H5Screate_simple_f ( 2, (/ncols, nrows/), mspc, h5err ) >> >> ! Fill in data variables >> do i = 1, ncols >> do j = 1, nrows >> matrix (i,j) = 100.0 * real(i) + real(j) >> end do >> end do >> >> ! Write dataspace in a loop >> ! New data has to be appended to the dataset already in the file. >> >> do i = 1, 30 >> print *, "Loop", i >> ! Get a new pointer to a copy of the dataspace for this set >> call H5Dget_space_f ( dset, dspc, h5err ) >> ! Figure out how big is the dataset in the file. >> call H5Sget_simple_extent_dims_f ( dspc, dimd, maxdim, h5err ) >> ! Extend size of the dataset in the file >> call H5Dextend_f ( dset, dimd+(/ncols,0/), h5err ) >> ! Close the dataspace now that it has been extended >> call H5Sclose_f ( dspc, h5err ) >> ! Fetch the dataspace of the extended dataset >> call H5Dget_space_f ( dset, dspc, h5err ) >> ! Select the appropriate slab in the extended dataset >> call H5Sselect_hyperslab_f ( dspc, H5S_SELECT_SET_F, & >> (/dimd(1),0/), (/1,1/), h5err, (/1,1/), shape(matrix) > ) >> ! Write the data from memory to the selected slab >> call H5Dwrite_f ( dset, H5T_NATIVE_REAL, matrix, shape(matrix), > >> h5err, & >> mspc, dspc, H5P_DEFAULT_F ) >> ! Close the extended dataspace to free memory >> call H5Sclose_f ( dspc, h5err ) >> call sleep (3) ! for a better view of process explorer >> graph >> end do >> >> ! Cleanup, end access and release resources >> >> call sleep (5) ! for a better view of process >> explorer graph >> call H5Dclose_f ( dset, h5err ) ! dataset >> !call H5Sclose_f ( dspc, h5err ) ! dataspace (released inside > loop) >> call H5Pclose_f ( dprp, h5err ) ! properties >> call H5Tclose_f ( dtyp, h5err ) ! data types >> >> call H5Fclose_f ( file, h5err ) ! file id >> call H5close_f ( h5err ) ! fortran interface >> call sleep (5) >> >> end program main >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > <nocompression.jpg><compression.jpg>__________________________________ > _____________ Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
