Re: [Hdf-forum] Select query options

Fri, 13 Aug 2010 10:59:29 -0700

Hi,

 HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

    Werner


On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <[email protected]> wrote:

Hi,

But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <[email protected]>
To: HDF Users Discussion List <[email protected]>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <[email protected]>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table.  Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted




--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Reply via email to