https://codeforge.lbl.gov/frs/?group_id=44
should do?
Werner
On Sat, 14 Aug 2010 07:00:57 +0200, Roy Mendelssohn <[email protected]>
wrote:
On the internet there is a reference to HDF5-FastQuery that uses FastBits
within HDF5 files. But finding the actual API has eluded me. Does anyone know
how to actually get this API?
Thanks,
-Roy M.
On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:
Hi,
Thats glad to know. I have explored fastbit, but fastbit requires that the file
be converted from hdf5 to binary file. With that in mind, would it be ideal to
just create the file in binary in the first place to reduce overhead. And use
fastbit itself instead of hdf5
Sent from my iPhone
On Aug 14, 2010, at 4:44 AM, Quincey Koziol <[email protected]> wrote:
Hi all,
On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:
Hi,
HDF5 doesn't support searching within datasets. You might want to look into
something like Fastbit:
http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html
This is an addon-library which builds indices on HDF5 datasets allowing to
select certain regions based on indexed properties.
We (The HDF Group) have just had a proposal funded that will integrate
FastBit [style] indexing more tightly with HDF5. The work is scheduled for a
little while from now (year 2 or 3 of the proposal, I believe), but I just
wanted to let people know that this functionality will be coming...
Quincey
Werner
On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <[email protected]> wrote:
Hi,
But that approach would be slow right? assuming that we are dealing
with the worst possible volume of data.
From: Francesc Alted <[email protected]>
To: HDF Users Discussion List <[email protected]>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options
2010/8/13 Rodel Piano <[email protected]>
Thanks for thr reply Francesc.
My codes are in C. Lets say my records are about 12million. I cant use
hyperslab coz i wont know which region to look for. What is the expected result
of my query is only 10 records out 12 million.
If you don't know which region you should look, then you should look into your
complete table. Just use hyperslabs for reading your dataset in small chunks
(my suggestion is that these chunks should fit your CPU cache, for improved
performance).
--
Francesc Alted
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
**********************
"The contents of this message do not reflect any position of the U.S. Government or
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: [email protected] (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
