Francesc, I do plan on using (or at least testing) the buffered file (H5FD_STDIO) for the hard drive based portion of my long-term storage. I also have a particular requirement to run in a disk-less environment for a much less amount of (current) data than I plan to store on the disk. I plan to periodically copying/moving the data from the memory file to the disk file by developing a server application responsible for its management.
I also have multiple clients which need access to the data and am planning on mirroring the memory-based portion of the data on each client's workstation with links to the network based disk storage (when available) for data beyond the memory file's capacity. Performance has been an issue in the legacy implementation for getting the data to the clients which is another reason I am experimenting with this configuration. I guess an alternative architecture would be do determine at runtime if a disk file was available and if not open a memory-based file and use it exclusively, otherwise exclusively use the networked hard drive... which is probably more in line with your suggestion. I plan to evaluate this as well. Of course, I am open to easier and/or better implementation ideas. Thanks for your input. Regards, Kirk -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Francesc Alted Sent: Tuesday, April 20, 2010 3:06 AM To: HDF Users Discussion List Subject: Re: [Hdf-forum] VFL: Combining Memory & Disk Files A Monday 19 April 2010 19:37:24 Kirk Harrison escrigué: > Does HDF5 support construction of a virtual HDF5 file composed of both > local disk (e.g., H5FD_SEC2, H5FD_STDIO) and memory-based files (e.g., > H5FD_CORE). possibly through the use of the H5FD_FAMILY driver? > > Application > > > HDF5 Virtual File Layer (VLF) > ( H5FD_FAMILY ) > > > H5FD_CORE H5FD_STDIO > > > Memory Hard Drive > > I also would like to create an HDF5 file that has a small and fast memory > cache for incoming live data and periodically write its contents to storage > for longer term retrieval. > > It is not clear to me through the documentation whether the H5FD_FAMILY is > only applicable to combining local disk (i.e. Hard Drive) based file > drivers. > > The goal that I'm shooting for is seamless access across both a > memory-based file and one that is stored on a hard disk. If I were you I would not bother too much in having different media for keeping your files: just take advantage of OS filesystem cache. When you have a small file that is accessed frequently, it is loaded in memory by the OS, so the access to it is made at memory speed, no disk speed. In addition, letting the OS to load in memory the data that is most accessed in your filesystem is probably the best way towards a sensible usage of computer resources. -- Francesc Alted _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
