Hello Thorben,
On Wed, 2010-03-03 at 08:08, Thorben Kröger wrote: > On Wednesday 03 March 2010 16:47:20 Mark Miller wrote: > > Hello Thorben, > > > > So, I am not sure I completely understood the part of this thread > > dealing with /dev/shm. However, did you in fact try using HDF5-1.8.4 and > > reading the file with the CORE vfd (H5Pset_fapl_core) > > I'm sorry, obviously I'm not deep into HDF5 yet or maybe I would have known > that function, but I didn't. I now read up on it > (for reference: http://*www.*hdfgroup.org/HDF5/doc/UG/08_TheFile.html ) > and I will try this out now... No worries. The interface has A LOT of options. > > > and setting your > > allocation inc to say 32*(1<<32) (thats 32 gig and just a little bigger > > than your file) > > What's an allocation inc? Well, I may have been a bit misleading here. Since you really only need to READ this file, then the allocation increment may not be that relevant. But, if you are creating or appending a file the allocation incriment tells the HDF5 library how much MORE memory to grab each time the 'file' (in memory) grows beyond the size of the currently allocated buffer. If you set this number too small, then you can wind up spending a lot of time in re-allocs (and subsequent memcpys) because you exceed the current size too often. If you set it too large, you can wind up allocating too much more memory than you need. The ability to READ file from the core vfd is a relatively new feature to HDF5. And, when you are READING, I am pretty sure the library is smart enough to set the initial buffer allocated to the size of the file that gets read. So, it may be that you actually DO NOT WANT to set the allocation increment to 32*(1<<32) as I suggested above but instead something quite small, say (1<<12) (4 kilobytes). Also, if you are ONLY reading the file, you probably want to set H5_ACC_RDONLY in the modes to the open call. Good luck. > > > and then opening the file with that fapl? I'd expect > > that to have helped a lot. > > I hope you're right; it will probably take me at least till tomorrow to get a > result, but will report back > > Cheers, > Thorben > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org -- Mark C. Miller, Lawrence Livermore National Laboratory ================!!LLNL BUSINESS ONLY!!================ [email protected] urgent: [email protected] T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
