Hi, IMHO, you should set your chunk size to a multiple of your filesystem stripe size. This way, each write should end up with the lowest possible writes.
Cheers, Matthieu 2013/9/18 Michael Jackson <[email protected]>: > I am developing some storage for a scientific instrument and am wanting to > get an idea of how to best "optimize" a chunk size. > > The basics of the setup are the following. The instrument collects an image > point by point. There are 10 different quantities at each point. most are > scalar quantities that are easy to deal with and pack into a single data set. > Their is one quantity that is actually another 2D image in of itself. The > size of this image can be as little as 80 x 60 all the way up to 1024 x 1024. > The instrument can "scan" an image of 2048 x 2048 in size. So to be clear I > am going to end up with a data set that is: > > 2048 x 2048 x 1024 x 1024 bytes in size (Worst case). > > My initial thought was to just chunk it by the 1024 x 1024 size which makes > striding through the data easy and natural for this application. Will having > that many chunks in a file impact the IO performance at some point? Are there > any general guidelines for setting the chunk size? > > Thanks > ___________________________________________________________ > Mike Jackson Principal Software Engineer > BlueQuartz Software Dayton, Ohio > [email protected] www.bluequartz.net > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/ _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
