Werner, Yes, they are linux-based. Interesting idea using a ramdisk for the short-term data. I had originally played with that concept, but somewhere in the midst of familiarizing myself with HDF5 picked up on the H5FD_CORE driver and forgot about it. :)
I'm not really concerned about an NSF mount to the ramdisk as I plan to have data flow directly to each application's host and written to a local memory-based (H5FD_CORE or ramdisk) file anyway. I need the data as close to processing-time as possible and only need the older data for less time critical historical recall. I.e., each client host will have its on constructed copy of a memory-based HDF5 (provided via an API that I am developing) containing very limited current data, with older data via a central file-based HDF5 data file (if available) accessed via an HDF5 link. The concept being that the API that I provide makes data queries across memory and disk based files transparent to the client. What are the trade-offs of using the H5FD_CORE driver versus a ramdisk? (Or is the H5FD_CORE basically implemented via ramdisk 'under the hood' anyway?) Regards, Kirk > Kirk, > > if your platform is linux-based, did you try creating your temporary > files in /dev/shm ? That's basically keeping them in RAM all time. > > I guess what you would like then is to export this ramdisk via NFS to > other > clients, but I doubt Linux can do remote NSFs mounting of ramdisks, though > maybe it's possible... > > Werner > > > On Tue, 20 Apr 2010 11:56:53 -0400, Kirk Harrison <[email protected]> > wrote: > >> Francesc, >> >> I do plan on using (or at least testing) the buffered file (H5FD_STDIO) >> for >> the hard drive based portion of my long-term storage. I also have a >> particular requirement to run in a disk-less environment for a much less >> amount of (current) data than I plan to store on the disk. I plan to >> periodically copying/moving the data from the memory file to the disk >> file >> by developing a server application responsible for its management. >> >> I also have multiple clients which need access to the data and am >> planning >> on mirroring the memory-based portion of the data on each client's >> workstation with links to the network based disk storage (when >> available) >> for data beyond the memory file's capacity. >> >> Performance has been an issue in the legacy implementation for getting >> the >> data to the clients which is another reason I am experimenting with this >> configuration. >> >> I guess an alternative architecture would be do determine at runtime if >> a >> disk file was available and if not open a memory-based file and use it >> exclusively, otherwise exclusively use the networked hard drive... which >> is >> probably more in line with your suggestion. I plan to evaluate this as >> well. >> >> Of course, I am open to easier and/or better implementation ideas. >> Thanks >> for your input. >> >> Regards, Kirk >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] >> On Behalf Of Francesc Alted >> Sent: Tuesday, April 20, 2010 3:06 AM >> To: HDF Users Discussion List >> Subject: Re: [Hdf-forum] VFL: Combining Memory & Disk Files >> >> A Monday 19 April 2010 19:37:24 Kirk Harrison escrigué: >>> Does HDF5 support construction of a virtual HDF5 file composed of both >>> local disk (e.g., H5FD_SEC2, H5FD_STDIO) and memory-based files (e.g., >>> H5FD_CORE). possibly through the use of the H5FD_FAMILY driver? >>> >>> Application >>> >>> >>> HDF5 Virtual File Layer (VLF) >>> ( H5FD_FAMILY ) >>> >>> >>> H5FD_CORE H5FD_STDIO >>> >>> >>> Memory Hard Drive >>> >>> I also would like to create an HDF5 file that has a small and fast >>> memory >>> cache for incoming live data and periodically write its contents to >> storage >>> for longer term retrieval. >>> >>> It is not clear to me through the documentation whether the H5FD_FAMILY >>> is >>> only applicable to combining local disk (i.e. Hard Drive) based file >>> drivers. >>> >>> The goal that I'm shooting for is seamless access across both a >>> memory-based file and one that is stored on a hard disk. >> >> If I were you I would not bother too much in having different media for >> keeping your files: just take advantage of OS filesystem cache. When >> you >> have >> a small file that is accessed frequently, it is loaded in memory by the >> OS, >> so >> the access to it is made at memory speed, no disk speed. In addition, >> letting >> the OS to load in memory the data that is most accessed in your >> filesystem >> is >> probably the best way towards a sensible usage of computer resources. >> > > > -- > ___________________________________________________________________________ > Dr. Werner Benger Visualization Research > Laboratory for Creative Arts and Technology (LCAT) > Center for Computation & Technology at Louisiana State University > (CCT/LSU) > 211 Johnston Hall, Baton Rouge, Louisiana 70803 > Tel.: +1 225 578 4809 Fax.: +1 225 578-5362 > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
