Kirk,
if your platform is linux-based, did you try creating your temporary files
in /dev/shm ? That's basically keeping them in RAM all time.
I guess what you would like then is to export this ramdisk via NFS to other
clients, but I doubt Linux can do remote NSFs mounting of ramdisks, though
maybe it's possible...
Werner
On Tue, 20 Apr 2010 11:56:53 -0400, Kirk Harrison <[email protected]> wrote:
Francesc,
I do plan on using (or at least testing) the buffered file (H5FD_STDIO) for
the hard drive based portion of my long-term storage. I also have a
particular requirement to run in a disk-less environment for a much less
amount of (current) data than I plan to store on the disk. I plan to
periodically copying/moving the data from the memory file to the disk file
by developing a server application responsible for its management.
I also have multiple clients which need access to the data and am planning
on mirroring the memory-based portion of the data on each client's
workstation with links to the network based disk storage (when available)
for data beyond the memory file's capacity.
Performance has been an issue in the legacy implementation for getting the
data to the clients which is another reason I am experimenting with this
configuration.
I guess an alternative architecture would be do determine at runtime if a
disk file was available and if not open a memory-based file and use it
exclusively, otherwise exclusively use the networked hard drive... which is
probably more in line with your suggestion. I plan to evaluate this as well.
Of course, I am open to easier and/or better implementation ideas. Thanks
for your input.
Regards, Kirk
-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of Francesc Alted
Sent: Tuesday, April 20, 2010 3:06 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] VFL: Combining Memory & Disk Files
A Monday 19 April 2010 19:37:24 Kirk Harrison escrigué:
Does HDF5 support construction of a virtual HDF5 file composed of both
local disk (e.g., H5FD_SEC2, H5FD_STDIO) and memory-based files (e.g.,
H5FD_CORE). possibly through the use of the H5FD_FAMILY driver?
Application
HDF5 Virtual File Layer (VLF)
( H5FD_FAMILY )
H5FD_CORE H5FD_STDIO
Memory Hard Drive
I also would like to create an HDF5 file that has a small and fast memory
cache for incoming live data and periodically write its contents to
storage
for longer term retrieval.
It is not clear to me through the documentation whether the H5FD_FAMILY is
only applicable to combining local disk (i.e. Hard Drive) based file
drivers.
The goal that I'm shooting for is seamless access across both a
memory-based file and one that is stored on a hard disk.
If I were you I would not bother too much in having different media for
keeping your files: just take advantage of OS filesystem cache. When you
have
a small file that is accessed frequently, it is loaded in memory by the OS,
so
the access to it is made at memory speed, no disk speed. In addition,
letting
the OS to load in memory the data that is most accessed in your filesystem
is
probably the best way towards a sensible usage of computer resources.
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
