Currently bloom filters and hfile indexes are in storefileIndexSize Why do you want bigger blocks? Just for less index size? Be aware it may lower random read performance.
J-D On Fri, Apr 15, 2011 at 8:23 AM, Matt Corgan <[email protected]> wrote: > Some of our servers have 5.2gb hbase heaps with the standard 40% (2gb) > memstore and 20% (1gb) blockcache. I'm wondering where the indexes and > bloom filters get counted. Is it in that remaining 2gb, and are the bloom > filters counted in the storefileIndexSize=566 metric? > > The overall data in each server isn't too large on disk (~50gb) after gzip > compression of 25x, but it's made of long keys with short values so there is > a lot of metadata. I'm thinking of upping the block size to 256k but > thought i'd ask how it worked first. > > some metrics: > > regions=401, stores=401, storefiles=758, storefileIndexSize=566, > memstoreSize=1793, usedHeap=3658, maxHeap=5183, blockCacheSize=816306600 > > Thanks for the help, > Matt > > On Fri, Apr 15, 2011 at 10:39 AM, Doug Meil > <[email protected]>wrote: > >> The HBase book has come through some updates recently on these metrics. >> >> http://hbase.apache.org/book.html#hbase_metrics >> >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf Of Stack >> Sent: Friday, April 15, 2011 5:04 AM >> To: [email protected] >> Cc: 陈加俊; [email protected] >> Subject: Re: heap memory allocation >> >> On Thu, Apr 14, 2011 at 9:50 PM, 陈加俊 <[email protected]> wrote: >> > How does heap memory allocated in regionserver ? >> > >> > Metrics: >> > >> > request=0.0, regions=1283, stores=2304, storefiles=1968, >> > storefileIndexSize=246, memstoreSize=791, compactionQueueSize=0, >> > usedHeap=5669, maxHeap=11991, blockCacheSize=2095157424, >> > blockCacheFree=419681872, blockCacheCount=24410, >> > blockCacheHitRatio=92, fsReadLatency=0, fsWriteLatency=0, >> > fsSyncLatency=0 >> > >> > >> > >> > maxHeap=11991 >> > usedHeap=5669 >> > >> >> This just the state of the JVM memory bean at the last update time. >> Looks like you have given hbase 12G. We're using about 5.5 Gig. See >> here for where we read it from the bean: >> >> http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/HRegionServer.html#751 >> These look like they report in MB. >> >> > blockCacheSize=2095157424 >> > memstoreSize=791 >> > ???? >> >> These look like they report in bytes and MB. Should make them all same >> units. HBase has a cache of hdfs blocks. And memstore is where we write >> edits too before they get flushed to the filesystem. >> >> > >> > Free=maxHeap-usedHeap? >> > >> >> Read the javadoc on the bean returned by >> ManagementFactory.getMemoryMXBean() call. >> >> St.Ack >> >> > >> > -- >> > Thanks & Best regards >> > jiajun >> > >> >
