Hi Matthieu, No, nothing different happens internally when handling larger datasets.
Some questions to understand more what you are doing: 1) what VFD are you using? I assume it's the MPIO VFD since you said HDF5 with parallel Processes, but you didn't say that explicitly. 2) Did you make sure your Lustre stripe settings (stripe count and size) are appropriate? i.e. the larger amount of data you write, the larger your stripe settings must be. Those are the only 2 things I can think of without looking at your code. Thanks, Mohamad -----Original Message----- From: Hdf-forum [mailto:[email protected]] On Behalf Of Matthieu Brucher Sent: Tuesday, January 21, 2014 2:25 PM To: HDF Users Discussion List Subject: [Hdf-forum] Performance hit when writing to big datasets Hi, I noticed today something strange. I've been using HDF5 with parallel processes, and when I raise the dataset size, the library seems to slow down. Currently, I have a fixed number of processes writing data from my simulation. All these processes write 1MB concurrently to contiguous but exclusive locations in the 1D dataset. When having more than 1 billion elements (for instance), the library is getting really, really slow (more than 20 times slower than what a usual pwrite would take), where as nothing can be noticed when writing say 1 million cell. Is there something fancy going on? I'm trying to access the simplest part of the library so that the H5write call can be more or less reduced to a pyrite call, but something is wrong. Should I set something to tell HF5 than it shouldn't do anything except than writing data from the calling process to the filesystem (Lustre in my case, MPI library configured with Lustre support). Regards, Matthieu Brucher -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher Music band: http://liliejay.com/ _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
