I was thinking it would be a nice feature if each time an hfile was written it kept a count of the raw bytes (before compression) to make it easy to compare to the file size on disk. It could report it in the web interface next to the disk size.
2011/5/25 Stack <[email protected]> > Good point Matt. I forgot about compression. Let me add not to the > above referenced section in the book.... > St.Ack > > On Wed, May 25, 2011 at 7:47 AM, Matt Corgan <[email protected]> wrote: > > I have a table that compresses by 30x using gzip, so the default block > size > > of 64 KB was writing 2 KB blocks to disk. To reduce storefileIndexSize, > I > > raised the block size to 256 KB, presumably writing ~8KB disk blocks > which > > is still pretty small. Maybe you could go even higher depending on your > > compression ratio. > > > > btw - why 10394 regions with only 3103 storefiles? > > > > > > 2011/5/25 Gaojinchao <[email protected]> > > > >> Region size is 512M > >> > >> hbase.regionserver.handler.count 50 > >> hbase.regionserver.global.memstore.upperLimit 0.4 > >> hbase.regionserver.global.memstore.lowerLimit 0.35 > >> hbase.hregion.memstore.flush.size 128M > >> hbase.hregion.max.filesize 512M > >> hbase.client.scanner.caching 1 hfile.block.cache.size 0.2 > >> hbase.hregion.memstore.block.multiplier 3 > >> hbase.hstore.blockingStoreFiles 10 > >> hbase.hstore.compaction.min.size 64M > >> > >> compress: gz > >> > >> dfs.block.size 256M > >> > >> -----邮件原件----- > >> 发件人: [email protected] [mailto:[email protected]] 代表 Stack > >> 发送时间: 2011年5月25日 11:57 > >> 收件人: [email protected] > >> 主题: Re: a question storefileIndexSize > >> > >> 2011/5/24 Gaojinchao <[email protected]>: > >> > Stack, Thanks for your reply. > >> > block size is default. > >> > My Key length is 26 bytes and value is 300~400 bytes. > >> > Is it big keys and small values ? > >> > > >> > >> Looks like you have 'small' keys. > >> > >> It looks like the index is about 1MB per storefile (storefiles=3103, > >> storefileIndexSize=3717). Does this seem about right? What size are > >> your regions? > >> > >> St.Ack > >> > > >
