On Wed, 2014-02-12 at 11:49 -0500, "André Walker-Loud " wrote: > Hi HDF-ers, > > I am working with people to implement the parallel hdf5 writer into some code > of ours, and I have a question regarding performance. > > The objects we are dealing with, as an array, look like > > [Nt,Nz,Ny,Nx,mu,a,b,…] > > where Nt is the slowest index. The code is parallelized amongst the first > four indices, and everything else is always stored locally on a node > (memory). The indices Nt,…,Nx are typically of size 32 - 128, while all the > other indices usually only are of size 3 or 4. It also happens that the > indices mu,a,b,… are typically flattened in memory to a single index. > > In terms of i/o performance, is there a recommendation for > > 1) writing an array [Nt,Nz,Ny,Nx,alpha], where alpha is a flattened array of > size Nmu x Na x Nb x … > > 2) unflatten alpha so the array is [Nt,Nz,Ny,Nx,mu,a,b,…]
Since mu, a, and b are small, my intuition is you should just store them in whatever way is most convenient for you. As long as you are using collective I/O -- and you have to ask for it explicitly via property list -- I'm pretty sure you'd be OK. Would be a good exercise to try out both approaches, see if there is a difference, then look at something like Darshan output to see why that difference might exist. ==rob _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
