I have a table that compresses by 30x using gzip, so the default block size of 64 KB was writing 2 KB blocks to disk. To reduce storefileIndexSize, I raised the block size to 256 KB, presumably writing ~8KB disk blocks which is still pretty small. Maybe you could go even higher depending on your compression ratio.
btw - why 10394 regions with only 3103 storefiles? 2011/5/25 Gaojinchao <[email protected]> > Region size is 512M > > hbase.regionserver.handler.count 50 > hbase.regionserver.global.memstore.upperLimit 0.4 > hbase.regionserver.global.memstore.lowerLimit 0.35 > hbase.hregion.memstore.flush.size 128M > hbase.hregion.max.filesize 512M > hbase.client.scanner.caching 1 hfile.block.cache.size 0.2 > hbase.hregion.memstore.block.multiplier 3 > hbase.hstore.blockingStoreFiles 10 > hbase.hstore.compaction.min.size 64M > > compress: gz > > dfs.block.size 256M > > -----邮件原件----- > 发件人: [email protected] [mailto:[email protected]] 代表 Stack > 发送时间: 2011年5月25日 11:57 > 收件人: [email protected] > 主题: Re: a question storefileIndexSize > > 2011/5/24 Gaojinchao <[email protected]>: > > Stack, Thanks for your reply. > > block size is default. > > My Key length is 26 bytes and value is 300~400 bytes. > > Is it big keys and small values ? > > > > Looks like you have 'small' keys. > > It looks like the index is about 1MB per storefile (storefiles=3103, > storefileIndexSize=3717). Does this seem about right? What size are > your regions? > > St.Ack >
