On Thu, Dec 09, 2010 at 10:57:28AM -0700, Leigh Orf wrote: > Thanks for the information. After I sent my email I realized I left out some > relevant information. I am not using pHDF5 but regular HDF5, but in a > parallel environment. The only reason I am doing this is because I want the > ability to write compressed HDF5 files (gzip, szip, scale-offset, nbit, > etc.). As I understand it, at this point (and maybe forever) pHDF5 cannot do > compression.
> I currently have tried two approaches with compression and HDF5 in a > parallel environment: (1) Each MPI rank writes its own compressed HDF5 file. > (2) I create a new MPI communicator (call it subcomm) which operates on a > sub-block of the entire domain. Each instance of subcomm (which could, for > instance, operate on one multicore chip) does a MPI_GATHER to rank 0 of > subcomm, and that root core does the compression and writes to disk. What if you still did collective writes with parallel-HDF5, but you did a little additional work in the application. If you compress each portion of data on each MPI rank, then ask HDF5 to write out that compressed buffer, blammo, you get parallel compression and parallel I/O. It's not as seamless as if you asked HDF5 to do the compression for you: I guess you'd have to find a stream-based compression algorithm (gzip?) that can work on concatenated blocks, and annotate the dataset with the compression algorithm you selected. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
