On Thu, Dec 9, 2010 at 11:40 AM, Rob Latham <[email protected]> wrote:
> On Thu, Dec 09, 2010 at 10:57:28AM -0700, Leigh Orf wrote: > > Thanks for the information. After I sent my email I realized I left out > some > > relevant information. I am not using pHDF5 but regular HDF5, but in a > > parallel environment. The only reason I am doing this is because I want > the > > ability to write compressed HDF5 files (gzip, szip, scale-offset, nbit, > > etc.). As I understand it, at this point (and maybe forever) pHDF5 cannot > do > > compression. > > > I currently have tried two approaches with compression and HDF5 in a > > parallel environment: (1) Each MPI rank writes its own compressed HDF5 > file. > > (2) I create a new MPI communicator (call it subcomm) which operates on a > > sub-block of the entire domain. Each instance of subcomm (which could, > for > > instance, operate on one multicore chip) does a MPI_GATHER to rank 0 of > > subcomm, and that root core does the compression and writes to disk. > > What if you still did collective writes with parallel-HDF5, but you > did a little additional work in the application. If you compress each > portion of data on each MPI rank, then ask HDF5 to write out that > compressed buffer, blammo, you get parallel compression and parallel > I/O. It's not as seamless as if you asked HDF5 to do the compression > for you: I guess you'd have to find a stream-based compression > algorithm (gzip?) that can work on concatenated blocks, and annotate > the dataset with the compression algorithm you selected. > I'd really like to be able to have HDF5 do the compression because I have grown quite accustomed to how transparent it is. The filters are just activated, and regardless of how you compress the data, you 'see' floating point data when you open the file or run h5dump or whatever. I could code things up to have each core open each hdf5 file, write each part of its file, close it, and hand it off to the next guy, but I just have to believe that's going to be really inefficient. It seems there should be a way to do this by passing file handles or property lists from one MPI process to another. I did find this page called "Collective HDF5 Calls in Parallel" which is interesting but it is unclear to me whether it applies to pHDF5 or just plain HDF5. Leigh > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > -- Leigh Orf Associate Professor of Atmospheric Science Department of Geology and Meteorology Central Michigan University Currently on sabbatical at the National Center for Atmospheric Research in Boulder, CO NCAR office phone: (303) 497-8200
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
