I just came upon this project, I was wondering if anyone has any experience with it good or bad:
STXXL: Standard Template Library for Extra Large Data sets http://stxxl.sourceforge.net/ Its goals are quite different from HDF5 certainly, but it seems to have more of a "paging engine" builtin than HDF5 does. So if I don't care so much about portability or sharing my data, but just want a fast runtime which can page, I wonder if this is worth considering. I'm open to building something simple on top of HDF5 as well, but just looking at options. -Philip On Wed, Dec 8, 2010 at 4:16 AM, Francesc Alted <[email protected]> wrote: > A Tuesday 07 December 2010 21:37:51 Philip Winston escrigué: > > > One final piece of warning: when you use mmap beyond the extend of > > > your RAM, you will end swapping out many data (shared libraries, > > > other processes) that might be important for the performance of > > > your computer. > > > > That is too bad you can't tell the OS to page-back your own file > > instead. That is if I have 24GB of RAM with 12GB available, and I > > mmap a 100GB file, I'd like it kind of churn through that 12GB of > > RAM with my stuff and leave everyone else alone. But I guess you > > don't have that control. > > Exactly, but I also think that you cannot control this. > > -- > Francesc Alted > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
