On Fri, Jul 15, 2011 at 10:06 AM, Claudio Martella <[email protected]> wrote: > On 7/15/11 6:24 PM, Stack wrote: >> How do you figure the N in the below Claudio? > N is the total amount of pairs in the sequence file. You know that when > you finish flushing a memstore or compacting files.
So a perfect index? If this is what you mean, won't the index often be often be close in size to the data size? (And index lives in memory at the moment; we have yet to make it so unused indices are let go if unused. And if this is what you mean, you can have a perfect index now by setting hfile block size < your smallest cell size. > That is interesting, the principle looks like the same as in HFile > (blocks and Btree) but I have to understand the difference further. > In HFile, the index is for every ~64k blocks or so; one block could have one entry only while the next block could have hundreds of entries; either has one entry in index only. So, its not the same as mapfile. It will write an index entry every 128 (IIRC) entries whether the entries are 10M each or 10bytes each (So, the indices could be wildly different in size based off cell size). >> IIRC, we have >> performance evaluation for various file types (You might be interested >> in this recent posting by Mikhail Bautin of an hfile v2). > I'd be interested in that, do you have a reference to it? > Here is Mikhail's post of an hfile v2 https://issues.apache.org/jira/browse/HBASE-3857 The HFile perf test tool is in code base at src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java (It was used comparing hfile vs tfile vs mapfile and then hfile tweaks; it may be a little stale). Thanks for bringing up this topic Claudio and for taking the time to do a bit of research. St.Ack
