On Mon, Dec 19, 2011 at 12:37:34PM -0600, John Knutson wrote: > Does anyone out there have any experience using rsync to copy HDF5 > files? I've been trying to use rsync to make back-ups of hdf5 files > as they grow, but instead of the expected fairly constant time > required for each update, the rsync time increases as the HDF5 file > grows. This suggests to me that rsync is re-transferring data > instead of just transferring differences. That, or as I add data to > the HDF5 file, changes are being made to numerous locations in the > file.
are you transferring locally, like to an external hard drive? rsync's -W (whole file) option says this: This [-W] is the default when both the source and destination are specified as local paths, but only if no batch-writing option is in effect. Does the '--no-whole-file' option help at all? ==rob > > I thought maybe the problem was that the time spent doing checksums > was causing the increase as the files grew in size, but the rsync > output indicates a linear increase in actual data transferred as > well, just like the run time. > > The files in question contain multiple data sets that are being > updated, each of which is stored as chunked, compressed data. > > The only thing I can think of to fiddle with on the rsync end is the > checksum block size, and try and make it more like the size of > blocks in the HDF5 file, which is an unknown to me at the moment. > > Alternately, I can make the files smaller, but that would not be my > first choice as it would be a major design change. > > If anyone has any suggestions as to how to resolve this "creeping > transfer time" issue, I'd appreciate it. > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
