Hi Quincey,
Thanks for your help!
I upgraded to HDF5 1.8.8 (src) as you suggested.
Now H5Pget_mpio_actual_io_mode() is called right after H5DWrite and
gives 0 ( = H5D_MPIO_NO_COLLECTIVE - return value is the second argument).
Why H5D_MPIO_NO_COLLECTIVE is returned I do not understand.
It is true that the data written by a process is non-continguous, can
this be the cause?
Best Regards,
Markus
On 2012-04-25 13:23, Quincey Koziol wrote:
Hi Markus,
Seems like it should be working, but can you upgrade to 1.8.8 (or the
1.8.9 prerelease) and use the H5Pget_mpio_actual_io_mode() routine to see if
collective I/O is occurring? Further actions will depend on the results from
that routine...
Quincey
On Apr 24, 2012, at 8:41 AM, Markus Bina wrote:
Dear HDF Forum users.
In my program I started using HDF5 1.8.2 and hyperslabs to write distributed
data to a single output file a few weeks ago.
The data is a Nx3 matrix (N is known at runtime).
What am I doing (see code below):
Opening a group. Creating a dataset for all the data. Each process selects
via hyperslabs where to write the data to (position).
The selected hyperslabs of two processes contain overlapping portions
(exactly the same data in the overlapping parts).
After that I do a collective write call. It is this call, which consumes a
lot of time (approx. 10 min for N=70e3).
Making the hypersalbs non-overlapping (takes less than a second) before does
not change the execution time significantly.
The program is being executed on a cluster having gpfs available and enabled
with mpirun and ~100 processes,
but might also be executed using other (non-parallel) file systems.
Does anyone have a hint for me?
Maybe I made a simple rookie mistake, which I just can't find.
I would not write this post if I had found the answer in manuals, tutorials or
via google.
Thanks!
Best Regards,
Markus
Code-snippet:
##################
group_id = H5Gopen(file_id, group_name.c_str(), H5P_DEFAULT);
dataspace_id = H5Screate_simple(2, dims, NULL);
dataset_id = H5Dcreate(group_id, HDFNamingPolicy::coord_data_name.c_str(),
H5T_NATIVE_DOUBLE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
status = H5Sclose(dataspace_id);
hid_t filespace = H5Dget_space(dataset_id);
slab_dims[0] = N; slab_dims[1] = 3;
dataspace_id = H5Screate_simple(2, slab_dims, NULL);
slab_dims[0] = 1 ;
for (int j = 0; j< N; j++)
{
offset[0] = this->_coord_map[j]; // figure out where to write the slab to
// write the row as hyperslab to file
// with partially overlapping data (the overlapping portions contain the
same numbers/data).
if (j==0) status = H5Sselect_hyperslab(filespace, H5S_SELECT_SET,
offset, NULL, slab_dims, NULL);
else status = H5Sselect_hyperslab(filespace, H5S_SELECT_OR, offset,
NULL, slab_dims, NULL);
}
hid_t plist_id = H5Pcreate(H5P_DATASET_XFER);
H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE);
// This is the slow part:
status = H5Dwrite(dataset_id, H5T_NATIVE_DOUBLE, dataspace_id, filespace,
plist_id, data);
status = H5Pclose(plist_id);
status = H5Sclose(dataspace_id);
H5Sclose(filespace);
status = H5Dclose(dataset_id);
status = H5Gclose(group_id);
#####################
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
