Hi John, Do you know which version of MPT you are running? NERSC received an updated version a few months back that fixed some problems with kernel I/O buffering for >4 threads on the XE6. You may want to double check that you have that update. 150MB/s is pretty bad. How many OSTs do you have and are you striping over all of them?
Mark On Tue, Apr 5, 2011 at 3:42 PM, Biddiscombe, John A. <[email protected]> wrote: >>> Have you tried using the MPI-POSIX VFD for independent access? > >>Thanks - I'll report back with further findings > > Rubbish! I still only get 150MB/s with the mpiposix driver. > > As Queen Victoria would have said "we are not amused" > > I suspect I've got an error somewhere because something should have changed. > > JB > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] > On Behalf Of Mark Howison > Sent: 05 April 2011 17:19 > To: HDF Users Discussion List > Subject: Re: [Hdf-forum] Chunking : was Howison et al Lustre mdc_config > > Hi John, > > What platform and parallel file system is this on? Have you tried > using the MPI-POSIX VFD for independent access? > > Thanks, > Mark > > On Tue, Apr 5, 2011 at 11:16 AM, Biddiscombe, John A. <[email protected]> > wrote: >> Elena, >> >> I was just replying to myself when your email came in. >> I knocked up a quick wrapper to enable testing of the cache eviction stuff >> so I'm happy (ish). >> >> However : I'm seeing puzzling behaviour with chunking. >> Using collective IO, (tweaking various params) I see transfer rates from >> 1GB/s to 3GB/s depending on stripe size, number of cb_nodes, etc. >> >> However, when using chunking with independent IO, I set stripe size to 6MB, >> chunk dims to match the 6MB, each node is writing 6MB, alignment is set to >> 6MB intervals and I've followed all the tips I can find. I see (for 512 >> nodes writing 6MB each = 4GB total) a max throughput of around 150MB/s. >> >> This is shockingly slow compared to collective IO and I'm quite surprised. >> I've been playing with this for a few days now and my general impression is >> that >> chunking = rubbish >> collective = nice >> >> I did not expect chunking to be so bad compared to collective (which is a >> shame as I was hoping to use it for compression etc). >> >> Can anyone suggest further tweaks that I should be looking out for to >> change. (one thing for example that seems to make no difference is the >> H5Pset_istore_k(fcpl, btree_ik); stuff. I still don't quite know what the >> correct value for btree_ik is. Ive read the man page, but I'm puzzled as to >> the correct meaning. if I know there will be 512 chunks, what is the 'right' >> value of btree_ik? >> >> Any clues gratefully received for optimizing chunking. I hoped the thread >> about 30,000 processes would carry on as I found it interesting to follow. >> >> ttfn >> >> JB >> >> >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] >> On Behalf Of Elena Pourmal >> Sent: 05 April 2011 17:01 >> To: HDF Users Discussion List >> Subject: Re: [Hdf-forum] Question re: Howison et al Lustre mdc_config tuning >> recommendations >> >> Hi John, >> On Apr 5, 2011, at 4:21 AM, Biddiscombe, John A. wrote: >> >>>>>> H5AC_cache_config_t mdc_config; >>>>>> hid_t file_id; >>>>>> file_id = H5Fopen("file.h5", H5ACC_RDWR, H5P_DEFAULT); >>>>>> mdc_config.version = H5AC__CURR_CACHE_CONFIG_VERSION; >>>>>> H5Pget_mdc_config(file_id, &mdc_config) >>>>>> mdc_config.evictions_enabled = 0 /* FALSE */; >>>>>> mdc_config.incr_mode = H5C_incr__off; >>>>>> mdc_config.decr_mode = H5C_decr__off; >>>>>> H5Pset_mdc_config(file_id, &mdc_config); >>> >>> I couldn't find fortran bindings for these. Do they exist in any recent >>> releases or svn branches. >>> >> Fortran wrappers for do not exist. Please let us know which Fortran wrappers >> do you need and we will add them to our to-do list. >> >> Elena >>> thanks >>> >>> JB >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
