Hi Pedro, Ok, I am not sure I fully understand what you are proposing or requesting but I certainly won't let that stop me from sharing my opinions ;)
It sounds like you are dealing with the fundamental 'read-modify-write' performance problems that often occurs in caching algorithms where operations 'above' the cache span multiple cache blocks. If I read your document correctly, it proposes a couple of solutions. One is to add your own temporary storage layer to act as a sort of impedance matcher thereby enforcing that any actual I/O requests to HDF5 match on cache block boundaries. The other solution involves a 'new API' I think. Is this a proposed new API for HDF5 Lib or HDF5 HL or HDF5 lite interfaces? Is the main issue that you need to be able to adjust chunk cache size based on your application's needs at the time the data is being read or written? If so, I think the HDF5 library has all the functions you'd need to do that already. So, its really not clear to me what value added you are proposing. If your datasets are 1 dimensional and you are processing more or less sequentially through it, as your pictures suggest, then I'd think a cache size equal to a few blocks should be sufficient to avoid the read-modify-write behavior. If its 2D, and your access is more or less a sliding 2D window, then I'd think a cache size of about 9 blocks would be sufficient to avoid read-modify-write behavior. If its 3D, then 27 blocks. So, I figure I must be missing something that motivates this because beyond manipulating the cache size, I am not seeing what else your 'new API' solution provides. Sorry if my response is off target. Mark On Mon, 2011-07-25 at 14:36 -0700, Pedro Silva Vicente wrote: > > Dear HDF community > > Please find a document detailing an HDF5 API develop at ORNL regarding > chunk caching. > > At this moment I would be very happy in receiving comments/questions/ > suggestions. > > > > ---------------------- > Pedro Vicente > [email protected] > http://www.space-research.org/ > > > > > _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
