On Fri, Apr 15, 2011 at 8:23 AM, Matt Corgan <[email protected]> wrote: > Some of our servers have 5.2gb hbase heaps with the standard 40% (2gb) > memstore and 20% (1gb) blockcache. I'm wondering where the indexes and > bloom filters get counted. Is it in that remaining 2gb, and are the bloom > filters counted in the storefileIndexSize=566 metric? >
Indexes are the storefileIndexSize metric. I don't think we count bloom space or rather, I believe they are kept in the cache (someone correct me if I'm off). > The overall data in each server isn't too large on disk (~50gb) after gzip > compression of 25x, but it's made of long keys with short values so there is > a lot of metadata. I'm thinking of upping the block size to 256k but > thought i'd ask how it worked first. > This skew -- long keys and short values -- makes for bigger indices for sure. Marc Limotte in an earlier thread takes a look at this indexing sizing. At first it seemed like the math was off and then he looked at his storefiles and figured that it starts to add up . See http://hbase.apache.org/book/keysize.html St.Ack
