Ha.. I bet that writeMetaDataDataset is the culprit here... so you are saying that you create a scalar dataset (with 1 element), and then write that same element n (n being the number of processes) the same time from all processes? Why would you need to do such a thing in the first place? If you need to write that element, you should just call writeMetaDataDataset from rank 0. If you don't need that float, then you should just not write it at all.
You called the metadata dataset an empty dataset essentially, so I understand that you don't need it? If that is the case, then why not create the attributes on the root group, or a different sub group for the current run, or even the large dataset? What you are causing is having every process grab a lock on that file system OST block, write that element and then release the lock. This is happening 960 times in your case, which I interpret what is causing this performance degradation.. Thanks, Mohamad -----Original Message----- From: Hdf-forum [mailto:[email protected]] On Behalf Of Maxime Boissonneault Sent: Wednesday, January 28, 2015 9:11 AM To: HDF Users Discussion List Subject: Re: [Hdf-forum] File keeps being updated long after the dataset is closed Le 2015-01-27 16:14, Mohamad Chaarawi a écrit : > Hi Maxime, > > If what I understand is true from what you are saying, you are getting pretty > good performance when writing the large dataset, but there is something else > going on that is slowing your I/O at the end? Yes. I am wondering if it is a matter of caching (i.e. HDF5 is waiting for a lock to come back, which would come back only after the data is actually written), or if it is what I do after writing the main dataset. > > Let me first mention that HDF5 manages in the background a metadata cache > (not to be confused with your metadata dataset). Any updates to the HDF5 file > may trigger a flush of the metadata cache at some point in time (usually it's > at file close if you are not doing a lot of metadata updates). The metadata > is internal to HDF5 and contains information about the file like object > header information, the file superblock, and many other things that are > transparent to the application.. This explains why you are seeing I/Os after > your dataset is closed. You will stop seeing updates on the file after you > call H5Fclose(). > The small "metadata" dataset is something I don't understand. What do you > mean by "it contains various data types"? The dataset is created with only 1 > HDF5 datatype and can't have multiple datatypes. Variable length datatypes > are not permitted in parallel. So please explain more what you mean by that. > Also how large is the small dataset and how are you writing to it? Do all 960 > MPI ranks write different hyperslabs to this small dataset (I wouldn't > imagine it would be small then), or does only 1 rank write your metadata to > that dataset? Are you using collective I/O if all processes are writing? Can > you use an attribute instead of a small dataset? The "metadata" dataset is created with H5Dcreate(...,H5T_NATIVE_FLOAT,...) It is being kept opened for the whole duration of the run, and has some attributes set on it, before being written. I guess that qualifies it as an empty dataset with multiple attributes. All ranks are calling the set attribute functions. Here is the code : http://pastebin.com/sq5ygQyM The functions createMetaDataDataset, setProperty, writeMetaDataDataset and closeMetaDataDataset are being called by all ranks. > It would be great if you can share the application so we can try it out, but > I understand that it might not always be possible. I think the above pastebin is all the code that you need, but I can paste the code for the whole class if you want. The code for the whole application will be problematic, not because of the code itself, but because of the data required as input. Maxime _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
