On Wed, 2011-02-23 at 04:54, Biddiscombe, John A. wrote: > Mark : Multi-pass VFD. I like this too.
I used the phrase 'two-pass planning VFD' primarily to jar Quincey's memory of a discussion we had a few weeks ago. However, the behavior I proposed is actually a change to HDF5 lib internal running on top of MPI-IO VFD. Since gap between CPU and I/O bandwidth is so wide and only getting wider, I see know problem with doing all the H5Dwrite work between H5Dopen and H5Dclose twice; once to get sizing information but not actually do any I/O and the second to then proceed with actual I/O given known sizing information. Now, I have conceived of a VFD that could be used to affect same behavior over a WHOLE FILE instead of an individual dataset. But, its only a glimmer in my eye right now ;) At any rate, either of these ideas does involve changes to application to essentially tell HDF5 twice what it wants to do. > It potentially allows a very flexible approach where even if > collective IO is writing to the same chunk, the collection/compression > phase can do the sums and transmit the info into the hdf5 metadata > layer. We'd certainly need to extend the chunking interface to handle > variable seized chunks to allow for more/less compression in different > areas of the data (actually this would be true for any option > involving lossless compression). I think the chunk hashing relies on > all chunks being the same size, so any change to that is going to be a > huge compatibility breaker. Also, the chunking layer sits on top of > the VFD, so I'm not sure if the VFD would be able to manipulate the > chunks in the way desired. Perhaps I'm mstaked and the VFD does see > the chunks. Correct me anyway. > > > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > [email protected] > > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org -- Mark C. Miller, Lawrence Livermore National Laboratory ================!!LLNL BUSINESS ONLY!!================ [email protected] urgent: [email protected] T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
