On Sat, May 11, 2013 at 10:31:05AM +0530, Suman Vajjala wrote: > 1.) What is the best way of organizing data in the file? I am planning to > have just 3 simple datasets for vertices, faces and cells. Should the > layout be 1-dimensional or chunked? The data is contiguous and there is no > contention among different processors during the write phase. Should I > worry about queuing I/O calls upto a limit and then flush the data. Should > I also worry about alignment issues (respecting stripe boundaries)?
My advice is predicated on using HDF5's MPI-IO driver and turning on collective i/o through an HDF5 property list. If the data is contiguous and there is no contention, then you can write however you like just about. The HDF5 library and the MPI-IO implementation should deal with file alignment. Are you going to write the data often (like a checkpoint)? > 2.) My next question is how do I efficiently read the data in an > unstructured manner from the file? The data to be read is not contiguous > and there is contention among processors? I am, presently, using > H5S_select_elements to do this. Is there a better way to do this? Will the > HDF5 API buffer data upto a limit and then transfer to memory? Are you going to read this data lots and lots of times? or is your post-simulation analysis a one-time thing? Optimize for the more frequent case, of course. If you are write-often, read-once, then your layout seems fine. Turn on MPI-IO and collective I/O and it matters a lot less how data is laid out. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
