When I thought about it, I didn't think cross-block compression would be a good idea - this is because you want to be able to decompress each block independently of each other. Perhaps a master HFile dictionary or something.
-ryan On Mon, Jun 6, 2011 at 12:06 AM, M. C. Srivas <[email protected]> wrote: > On Sun, Jun 5, 2011 at 11:37 PM, Ryan Rawson <[email protected]> wrote: > >> On Sun, Jun 5, 2011 at 11:33 PM, Jason Rutherglen >> <[email protected]> wrote: >> > Ok, the block index is only storing the first key of each block? >> > Hmm... I think we can store a pointer to an exact position in the >> > block, or at least allow that (for the FST implementation). >> >> Are you sure that is a good idea? Surely the disk seeks would destroy >> you on index load? >> > > I agree, it would be pretty bad. > > But, assuming that the block size is set appropriately, copying one key per > 100 or so values into the block index does not really bloat the hfile and is > good trade-off to avoid the seeking. Plus, it does not prevent > prefix-compression inside the block itself. Are we considering > prefix-compression of keys across blocks? > > > >> >> >> > >> > How efficient is the current seeking? >> > >> >> I have previously thought about prefix compression, it seemed doable, >> > >> > It does look like prefix compression should be doable. Eg, we'd seek >> > to a position based on the block index (from which we'd have the >> > entire key). From the seek'd to position, we could scan and load up >> > each subsequent prefix compressed key into a KeyValue, though right >> > the KV wouldn't be 'pointing' back to the internals of the block, it'd >> > be creating a whole new byte[] for each KV (which could have it's own >> > garbage related ramifications). >> > >> >> you'd need a compressing algorithm >> > >> > Lucene's terms dict is very simple. The next key has the position at >> > which the previous key differs. >> > >> > On Sat, Jun 4, 2011 at 3:35 PM, Ryan Rawson <[email protected]> wrote: >> >> Also, dont break it :-) >> >> >> >> Part of the goal of HFile was to build something quick and reliable. >> >> It can be hard to know you have all the corner cases down and you >> >> won't find out in 6 months that every single piece of data you have >> >> put in HBase is corrupt. Keeping it simple is one strategy. >> >> >> >> I have previously thought about prefix compression, it seemed doable, >> >> you'd need a compressing algorithm, then in the Scanner you would >> >> expand KeyValues and callers would end up with copies, not views on, >> >> the original data. The JVM is fairly good about short lived objects >> >> (up to a certain allocation rate that is), and while the original goal >> >> was to reduce memory usage, it could make sense to take a higher short >> >> term allocation rate if the wins from prefix compression are there. >> >> >> >> Also note that in whole-system profiling, often repeated methods in >> >> KeyValue do pop up. The goal of KeyValue was to have a format that >> >> didnt require deserialization into larger data structures (hence the >> >> lack of vint), and would be simple and fast. Undoing that work should >> >> be accompanied with profiling evidence that new slowdowns were not >> >> introduced. >> >> >> >> -ryan >> >> >> >> On Sat, Jun 4, 2011 at 3:30 PM, Jason Rutherglen >> >> <[email protected]> wrote: >> >>>> You'd have to change how the Scanner code works, etc. You'll find >> out. >> >>> >> >>> Nice! Sounds fun. >> >>> >> >>> On Sat, Jun 4, 2011 at 3:27 PM, Ryan Rawson <[email protected]> >> wrote: >> >>>> What are the specs/goals of a pluggable block index? Right now the >> >>>> block index is fairly tied deep in how HFile works. You'd have to >> >>>> change how the Scanner code works, etc. You'll find out. >> >>>> >> >>>> >> >>>> >> >>>> On Sat, Jun 4, 2011 at 3:17 PM, Stack <[email protected]> wrote: >> >>>>> I do not know of one. FYI hfile is pretty standalone regards tests >> etc. There is even a perf testing class for hfile >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Jun 4, 2011, at 14:44, Jason Rutherglen < >> [email protected]> wrote: >> >>>>> >> >>>>>> I want to take a wh/hack at creating a pluggable block index, is >> there >> >>>>>> an open issue for this? I looked and couldn't find one. >> >>>>> >> >>>> >> >>> >> >> >> > >> >
