On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?
Thanks, -Roy M. On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote: > Hi, > > Thats glad to know. I have explored fastbit, but fastbit requires that the > file be converted from hdf5 to binary file. With that in mind, would it be > ideal to just create the file in binary in the first place to reduce > overhead. And use fastbit itself instead of hdf5 > > Sent from my iPhone > > On Aug 14, 2010, at 4:44 AM, Quincey Koziol <[email protected]> wrote: > >> Hi all, >> >> On Aug 13, 2010, at 12:57 PM, Werner Benger wrote: >> >>> Hi, >>> >>> HDF5 doesn't support searching within datasets. You might want to look >>> into something like Fastbit: >>> >>> http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html >>> http://crd.lbl.gov/~kewu/fastbit/doc/index.html >>> >>> This is an addon-library which builds indices on HDF5 datasets allowing to >>> select certain regions based on indexed properties. >> >> We (The HDF Group) have just had a proposal funded that will integrate >> FastBit [style] indexing more tightly with HDF5. The work is scheduled for >> a little while from now (year 2 or 3 of the proposal, I believe), but I just >> wanted to let people know that this functionality will be coming... >> >> Quincey >> >>> Werner >>> >>> >>> On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <[email protected]> >>> wrote: >>> >>> Hi, >>> >>> But that approach would be slow right? assuming that we are dealing >>> with the worst possible volume of data. >>> >>> From: Francesc Alted <[email protected]> >>> To: HDF Users Discussion List <[email protected]> >>> Sent: Fri, August 13, 2010 10:30:56 PM >>> Subject: Re: [Hdf-forum] Select query options >>> >>> 2010/8/13 Rodel Piano <[email protected]> >>> Thanks for thr reply Francesc. >>> >>> My codes are in C. Lets say my records are about 12million. I cant use >>> hyperslab coz i wont know which region to look for. What is the expected >>> result of my query is only 10 records out 12 million. >>> >>> If you don't know which region you should look, then you should look into >>> your complete table. Just use hyperslabs for reading your dataset in small >>> chunks (my suggestion is that these chunks should fit your CPU cache, for >>> improved performance). >>> >>> -- >>> Francesc Alted >>> >>> >>> >>> >>> -- >>> ___________________________________________________________________________ >>> Dr. Werner Benger Visualization Research >>> Laboratory for Creative Arts and Technology (LCAT) >>> Center for Computation & Technology at Louisiana State University (CCT/LSU) >>> 211 Johnston Hall, Baton Rouge, Louisiana 70803 >>> Tel.: +1 225 578 4809 Fax.: +1 225 578-5362 >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org ********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: [email protected] (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
