Re: Pluggable block index

2011-06-06 Thread Jason Rutherglen
Ok, the block index is only storing the first key of each block? Hmm... I think we can store a pointer to an exact position in the block, or at least allow that (for the FST implementation). How efficient is the current seeking? I have previously thought about prefix compression, it seemed

Re: Pluggable block index

2011-06-06 Thread Ryan Rawson
On Sun, Jun 5, 2011 at 11:33 PM, Jason Rutherglen jason.rutherg...@gmail.com wrote: Ok, the block index is only storing the first key of each block? Hmm... I think we can store a pointer to an exact position in the block, or at least allow that (for the FST implementation). Are you sure that

Re: Pluggable block index

2011-06-06 Thread M. C. Srivas
On Sun, Jun 5, 2011 at 11:37 PM, Ryan Rawson ryano...@gmail.com wrote: On Sun, Jun 5, 2011 at 11:33 PM, Jason Rutherglen jason.rutherg...@gmail.com wrote: Ok, the block index is only storing the first key of each block? Hmm... I think we can store a pointer to an exact position in the

Re: Pluggable block index

2011-06-06 Thread Ryan Rawson
When I thought about it, I didn't think cross-block compression would be a good idea - this is because you want to be able to decompress each block independently of each other. Perhaps a master HFile dictionary or something. -ryan On Mon, Jun 6, 2011 at 12:06 AM, M. C. Srivas mcsri...@gmail.com

Re: Pluggable block index

2011-06-06 Thread Jason Rutherglen
But, assuming that the block size is set appropriately, copying one key per 100 or so values into the block index does not really bloat the hfile Right, this is what Lucene does and should work fine for HBase. Eg, the FST will enable a denser index of keys at the same storage costs.

Re: running unit tests

2011-06-06 Thread Stack
On TRUNK, usually all tests pass but intermittently some are broken. The best gauge for the current state of trunk can be found here, https://builds.apache.org//job/HBase-TRUNK/, on the Apache CI server. As J-D says, there is nothing special that you need to do to run the tests other than 'mvn

RE: HBase Push Video or Image File

2011-06-06 Thread sgtpepper
Hey Doug, Thanks a lot for the reply. But is there any sample/example code available that does the same? I found this code in code.google.com, but it looks more like it is putting the avi file in HDFS instead of HBase. http://code.google.com/p/hbase-jdo/ But I cant seem to be able to understand

RE: HBase Push Video or Image File

2011-06-06 Thread Doug Meil
Hi there- I'd recommend reading this... http://hbase.apache.org/book.html#datamodel ... and focusing on what is available in the HBase client API before worrying about hbase-util, which are wrappers on the same HBase client API. -Original Message- From: sgtpepper