Hi Malcolm, On Sep 19, 2012, at 1:23 PM, Malcolm MacLeod wrote:
> Hello Elena, >> Just occurred to me....Did you check that the program closes unused >> identifiers? This may cause performance degrade. > Yes, I have checked for this and did not see anything. > Some further information that I have uncovered, on Windows 7 this degredation > does not occur but on Windows XP it does, so seemingly whatever is going on > here is XP specific? > > I will see if I can find some time to make a test case. > This will be great! Please send it to [email protected] Thank you! Elena > Thanks, > Malcolm > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Elena Pourmal The HDF Group http://hdfgroup.org >> 1800 So. Oak St., Suite 203, Champaign IL 61820 >> 217.531.6112 >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> On Sep 8, 2012, at 3:31 AM, Malcolm MacLeod wrote: >>> Hello Elena, >>> >>> Sorry I should have mentioned that, I am already setting H5F_LIBVER_LATEST >>> and have recreated the file (which is what gave the slight speed boost I >>> mentioned originally when upgrading) but the same issue is unfortunately >>> still present. >>> >>> - Malcolm >>> >>>> Malcolm, >>>> >>>> Please try to use the latest file format when you create a file. It >>>> should >>>> be more efficient in handling groups with a big number of objects. >>>> >>>> See the H5Pset_libver_bounds function >>>> (http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetLibverBounds >>>> ); >>>> use H5F_LIBVER_LATEST for the last two parameters. >>>> >>>> You may repack an existing file with h5repack using -L flag. >>>> >>>> Elena >>>> >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> Elena Pourmal The HDF Group http://hdfgroup.org >>>> 1800 So. Oak St., Suite 203, Champaign IL 61820 >>>> 217.531.6112 >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> >>>> On Sep 5, 2012, at 4:25 AM, Malcolm MacLeod wrote: >>>>> Hello, >>>>> >>>>> Our software has for a long time made use of the HDF5 library without >>>>> any >>>>> issues. Recently we have started to run into datasets far larger than wh >>>>> at >>>>> was previously used and some scalability issues appear to be showing. >>>>> >>>>> The HDF5 file in question contains a single group with many datasets - A >>>>> specific piece of code opens every dataset one at a time and reads from >>>>> it >>>>> via H5DRead. >>>>> >>>>> Previously it was rare to have more than ~90000 datasets here so this >>>>> was >>>>> never noticed - but after H5DRead has been called about ~60000 times >>>>> subsequent calls appear to start to become increasingly slow, by about >>>>> ~80000 calls it slows to a crawl (instead of processing 1000s a second >>>>> it >>>>> is processing only two or three per second) >>>>> >>>>> I have tried upgrading from 1.8.8 -> 1.8.9 and this seems to have helped >>>>> slightly, it now becomes unbearable at around ~100000 instead of ~80000 >>>>> calls. >>>>> >>>>> >>>>> Some observations: >>>>> 1) This does not appear to be due to a seek delay or (larger datasets in >>>>> the middle) or anything like that, I have tried e.g. starting at the >>>>> back >>>>> of a group of ~500000 datasets instead of the front and the same thing >>>>> happens. I have tried also to start in various spots towards the middle >>>>> and also the same behaviour can be observed. >>>>> 2) If I cancel the loop, allow the software to idle for a while and then >>>>> give it another go the same thing happens (it is fast again until a >>>>> certain quantity of reads) - so it appears that HDF5 may be doing >>>>> something in the background once it is not busy that allows reads to be >>>>> fast again? >>>>> >>>>> >>>>> I would greatly appreciate any thoughts on this or ideas as to what >>>>> might >>>>> be going on? >>>>> >>>>> Regards, >>>>> Malcolm MacLeod >>>>> >>>>> _______________________________________________ >>>>> Hdf-forum is for HDF software users discussion. >>>>> [email protected] >>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
