I am developing some storage for a scientific instrument and am wanting to get an idea of how to best "optimize" a chunk size.
The basics of the setup are the following. The instrument collects an image point by point. There are 10 different quantities at each point. most are scalar quantities that are easy to deal with and pack into a single data set. Their is one quantity that is actually another 2D image in of itself. The size of this image can be as little as 80 x 60 all the way up to 1024 x 1024. The instrument can "scan" an image of 2048 x 2048 in size. So to be clear I am going to end up with a data set that is: 2048 x 2048 x 1024 x 1024 bytes in size (Worst case). My initial thought was to just chunk it by the 1024 x 1024 size which makes striding through the data easy and natural for this application. Will having that many chunks in a file impact the IO performance at some point? Are there any general guidelines for setting the chunk size? Thanks ___________________________________________________________ Mike Jackson Principal Software Engineer BlueQuartz Software Dayton, Ohio [email protected] www.bluequartz.net _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
