Le 2015-01-27 16:14, Mohamad Chaarawi a écrit :
Hi Maxime,
If what I understand is true from what you are saying, you are getting pretty
good performance when writing the large dataset, but there is something else
going on that is slowing your I/O at the end?
Yes. I am wondering if it is a matter of caching (i.e. HDF5 is waiting
for a lock to come back, which would come back only after the data is
actually written), or if it is what I do after writing the main dataset.
Let me first mention that HDF5 manages in the background a metadata cache (not
to be confused with your metadata dataset). Any updates to the HDF5 file may
trigger a flush of the metadata cache at some point in time (usually it's at
file close if you are not doing a lot of metadata updates). The metadata is
internal to HDF5 and contains information about the file like object header
information, the file superblock, and many other things that are transparent to
the application.. This explains why you are seeing I/Os after your dataset is
closed. You will stop seeing updates on the file after you call H5Fclose().
The small "metadata" dataset is something I don't understand. What do you mean by
"it contains various data types"? The dataset is created with only 1 HDF5 datatype and
can't have multiple datatypes. Variable length datatypes are not permitted in parallel. So please
explain more what you mean by that.
Also how large is the small dataset and how are you writing to it? Do all 960
MPI ranks write different hyperslabs to this small dataset (I wouldn't imagine
it would be small then), or does only 1 rank write your metadata to that
dataset? Are you using collective I/O if all processes are writing? Can you use
an attribute instead of a small dataset?
The "metadata" dataset is created with
H5Dcreate(...,H5T_NATIVE_FLOAT,...)
It is being kept opened for the whole duration of the run, and has some
attributes set on it, before being written. I guess that qualifies it as
an empty dataset with multiple attributes. All ranks are calling the set
attribute functions. Here is the code :
http://pastebin.com/sq5ygQyM
The functions createMetaDataDataset, setProperty, writeMetaDataDataset
and closeMetaDataDataset are being called by all ranks.
It would be great if you can share the application so we can try it out, but I
understand that it might not always be possible.
I think the above pastebin is all the code that you need, but I can
paste the code for the whole class if you want. The code for the whole
application will be problematic, not because of the code itself, but
because of the data required as input.
Maxime
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
