Thanks.. I've read the pertinent sections and what I'm coming away with
is that the chunk *sizes* should be designed around the I/O bandwidth of
your disk subsystem, and the *shapes* should be designed around the
access patterns for the data and around the data set itself (avoid
mostly empty chunks and so on, as per 5.1.2 guidelines)...
What this doesn't really get into, it seems to me, is the role of the
raw data chunk cache in all of this.
I don't think contiguous data is even an option for us, as we would have
several multi-terabyte data sets which take quite some time just to
initialize on disk.
Ruth Aydt wrote:
You may find some of the chunking discussions in this paper of interest:
http://www.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf
in particular, section 3.2 and port6ions of sections 4 & 5.
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
