Hi, Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5
Sent from my iPhone On Aug 14, 2010, at 4:44 AM, Quincey Koziol <[email protected]> wrote: > Hi all, > > On Aug 13, 2010, at 12:57 PM, Werner Benger wrote: > >> Hi, >> >> HDF5 doesn't support searching within datasets. You might want to look into >> something like Fastbit: >> >> http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html >> http://crd.lbl.gov/~kewu/fastbit/doc/index.html >> >> This is an addon-library which builds indices on HDF5 datasets allowing to >> select certain regions based on indexed properties. > > We (The HDF Group) have just had a proposal funded that will integrate > FastBit [style] indexing more tightly with HDF5. The work is scheduled for a > little while from now (year 2 or 3 of the proposal, I believe), but I just > wanted to let people know that this functionality will be coming... > > Quincey > >> Werner >> >> >> On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <[email protected]> >> wrote: >> >> Hi, >> >> But that approach would be slow right? assuming that we are dealing >> with the worst possible volume of data. >> >> From: Francesc Alted <[email protected]> >> To: HDF Users Discussion List <[email protected]> >> Sent: Fri, August 13, 2010 10:30:56 PM >> Subject: Re: [Hdf-forum] Select query options >> >> 2010/8/13 Rodel Piano <[email protected]> >> Thanks for thr reply Francesc. >> >> My codes are in C. Lets say my records are about 12million. I cant use >> hyperslab coz i wont know which region to look for. What is the expected >> result of my query is only 10 records out 12 million. >> >> If you don't know which region you should look, then you should look into >> your complete table. Just use hyperslabs for reading your dataset in small >> chunks (my suggestion is that these chunks should fit your CPU cache, for >> improved performance). >> >> -- >> Francesc Alted >> >> >> >> >> -- >> ___________________________________________________________________________ >> Dr. Werner Benger Visualization Research >> Laboratory for Creative Arts and Technology (LCAT) >> Center for Computation & Technology at Louisiana State University (CCT/LSU) >> 211 Johnston Hall, Baton Rouge, Louisiana 70803 >> Tel.: +1 225 578 4809 Fax.: +1 225 578-5362 >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
