On Tue, 07 Dec 2010 21:59:41 +0100, Mark Miller <[email protected]> wrote:
On Tue, 2010-12-07 at 12:53 -0800, Werner Benger wrote:
I've been facing similar issues, like scanning through 500GB HDF5
files with 32GB available RAM.
That works ok with HDF5, but I've implemented my own memory management
strategy to tell
it which parts to keep in memory and which parts to unload (of course,
using random access to
the datasets). Turned out that the "remove the least-used-object"
strategy is not necessarily
the best one (as the OS would follow with pages), but some
classification on similarity of objects
that are kept or to be discarded from memory seems much more
efficient.
Out of curiosity, did you consider/need to use
posix_fadvise/posix_madvise to give the OS a hint that the application
is managing the memory/caching and so the OS should NOT attempt to?
(another way of doing this I guess is using 'direct I/O' -- O_DIRECT,
though that is not available very many places).
Mark
Hm no. I just used HDF5 as it is, as the application needs to be
cross-platform Linux, MacOS, Windows. Does the HDF5 API already support
using such hints? It seemed ok to have two levels of caching in my
case (such as application-internal and HDF5/OS internal), as the entire
system was supposed to stay at 30% overall RAM usage anyway to leave space
for other applications running on the same machine. Probably those direct
I/O
parameters could help to optimize this somewhat and make more efficient
usage of the available memory, but so far there was no urgent need for it.
Might be worth consideration on future improvements, though.
Werner
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
