Hi Leigh, On Dec 9, 2010, at 12:52 PM, Leigh Orf wrote:
> On Thu, Dec 9, 2010 at 11:40 AM, Rob Latham <[email protected]> wrote: > On Thu, Dec 09, 2010 at 10:57:28AM -0700, Leigh Orf wrote: > > Thanks for the information. After I sent my email I realized I left out some > > relevant information. I am not using pHDF5 but regular HDF5, but in a > > parallel environment. The only reason I am doing this is because I want the > > ability to write compressed HDF5 files (gzip, szip, scale-offset, nbit, > > etc.). As I understand it, at this point (and maybe forever) pHDF5 cannot do > > compression. > > > I currently have tried two approaches with compression and HDF5 in a > > parallel environment: (1) Each MPI rank writes its own compressed HDF5 file. > > (2) I create a new MPI communicator (call it subcomm) which operates on a > > sub-block of the entire domain. Each instance of subcomm (which could, for > > instance, operate on one multicore chip) does a MPI_GATHER to rank 0 of > > subcomm, and that root core does the compression and writes to disk. > > What if you still did collective writes with parallel-HDF5, but you > did a little additional work in the application. If you compress each > portion of data on each MPI rank, then ask HDF5 to write out that > compressed buffer, blammo, you get parallel compression and parallel > I/O. It's not as seamless as if you asked HDF5 to do the compression > for you: I guess you'd have to find a stream-based compression > algorithm (gzip?) that can work on concatenated blocks, and annotate > the dataset with the compression algorithm you selected. > > I'd really like to be able to have HDF5 do the compression because I have > grown quite accustomed to how transparent it is. The filters are just > activated, and regardless of how you compress the data, you 'see' floating > point data when you open the file or run h5dump or whatever. > > I could code things up to have each core open each hdf5 file, write each part > of its file, close it, and hand it off to the next guy, but I just have to > believe that's going to be really inefficient. It seems there should be a way > to do this by passing file handles or property lists from one MPI process to > another. We've been working on some changes that allow the metadata cache to be flushed and refreshed without re-opening the whole file. That might apply here, but we weren't thinking about concurrent processes writing to the file, so it might require further work to support that use case. We'd also have to update the file free space information, which would be more work. Hmm... I'll bounce it around with some of the DOE folks who may be able to help fund this, but it would be a good use case for getting some funding to support Blue Waters, also. > I did find this page called "Collective HDF5 Calls in Parallel" which is > interesting but it is unclear to me whether it applies to pHDF5 or just plain > HDF5. That applies to using HDF5 with the MPI-IO or MPI-POSIX VFDs (which implicitly requires a parallel program using MPI). Quincey > Leigh > > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > > > -- > Leigh Orf > Associate Professor of Atmospheric Science > Department of Geology and Meteorology > Central Michigan University > Currently on sabbatical at the National Center for Atmospheric Research in > Boulder, CO > NCAR office phone: (303) 497-8200 > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
