In HBase client, you can utilize HeapSize through Put:
public class Put implements HeapSize, Writable, Row, Comparable<Row> {
Cheers
On Wed, Aug 25, 2010 at 3:13 PM, Ryan Rawson <ryano...@gmail.com> wrote:
> Hey,
>
> It's all over the place. HeapSize is one interface. There is also
> stuff in HRegion, Store and Memstore that tracks memory used. On top
> of it the LruBlockCache does more stuff as well.
>
> -ryan
>
> On Wed, Aug 25, 2010 at 10:59 AM, Alex Baranau <alex.barano...@gmail.com>
> wrote:
> > Hello guys,
> >
> >> HBase is a self memory tuning app
> >
> > Is there any documentation on that or if you could point to some sources
> I
> > will be very thankful (even if sources are particular classes, please
> name
> > at least few to look at).
> >
> > Thank you in advance,
> > Alex Baranau.
> >
> > On Tue, Aug 10, 2010 at 1:25 AM, Ryan Rawson <ryano...@gmail.com> wrote:
> >
> >> Hey,
> >>
> >> HBase is a self memory tuning app, unlike most Java apps you may have
> >> experience with. We use as much memory as possible for memstore and
> >> block cache, but we also flush memory to keep overall usage below
> >> maximums.
> >>
> >> Enjoy,
> >> -ryan
> >>
> >> On Mon, Aug 9, 2010 at 3:22 PM, Jinsong Hu <jinsong...@hotmail.com>
> wrote:
> >> > you are right. now it shows up as
> >> >
> >> > request=180.5, regions=60, stores=61, storefiles=67,
> >> storefileIndexSize=143,
> >> > memstoreSize=20, compactionQueueSize=0, usedHeap=1417, maxHeap=6127,
> >> > blockCacheSize=1061944288, blockCacheFree=223059360,
> >> blockCacheCount=16239,
> >> > blockCacheHitRatio=14
> >> >
> >> > still 60 regions, and the usedHeap is only 1.4G. I already put all the
> >> > memory tunning suggested in the hbase-env.sh to do incremental GC.
> >> > looks the jvm still have drastic GC even with the incremental GC
> turned
> >> on.
> >> > now the GC log shows:
> >> >
> >> > 1462948.533: [GC 1462948.533: [ParNew: 138376K->1441K(153344K),
> 0.0037940
> >> > secs]
> >> > 1446591K->1309805K(6274432K) icms_dc=0 , 0.0038990 secs] [Times:
> >> user=0.03
> >> > sys=0
> >> > .00, real=0.00 secs]
> >> > 1463005.844: [GC 1463005.845: [ParNew: 137716K->17024K(153344K),
> >> 0.0139630
> >> > secs]
> >> > 1446081K->1327172K(6274432K) icms_dc=0 , 0.0140740 secs] [Times:
> >> user=0.11
> >> > sys=
> >> > 0.00, real=0.02 secs]
> >> >
> >> > which is reasonable.
> >> >
> >> > Jimmy.
> >> >
> >> > --------------------------------------------------
> >> > From: "Ryan Rawson" <ryano...@gmail.com>
> >> > Sent: Monday, August 09, 2010 3:07 PM
> >> > To: <user@hbase.apache.org>
> >> > Subject: Re: memory usage too high in regionserver
> >> >
> >> >> Unfortunately 'usedHeap' includes the unfound garbage...
> >> >>
> >> >> Perhaps you can post more of your GC log.
> >> >>
> >> >> On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <jinsong...@hotmail.com>
> >> wrote:
> >> >>>
> >> >>> Hi, There:
> >> >>>
> >> >>> I am using cloudera cdh3 regionserver and today, I noticed that one
> of
> >> >>> the
> >> >>> regionserver memory usage is very high:
> >> >>>
> >> >>> request=8.8, regions=60, stores=61, storefiles=67,
> >> >>> storefileIndexSize=142,
> >> >>> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
> >> >>> blockCacheSize=1061944288, blockCacheFree=223059360,
> >> >>> blockCacheCount=16239,
> >> >>> blockCacheHitRatio=14
> >> >>>
> >> >>> this shows that only 60 regions calused close to 6G of memory usage.
> >> This
> >> >>> is
> >> >>> very high.
> >> >>>
> >> >>> the gc-hbase.log shows:
> >> >>>
> >> >>> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K),
> >> >>> 0.0487050
> >> >>> secs]
> >> >>> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times:
> >> >>> user=0.33
> >> >>> sys=
> >> >>> 0.00, real=0.05 secs]
> >> >>> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K),
> >> >>> 0.0454440
> >> >>> secs]
> >> >>> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times:
> >> >>> user=0.31
> >> >>> sys=
> >> >>> 0.01, real=0.05 secs]
> >> >>>
> >> >>>
> >> >>> This shows that the regionserver is indeed using 6G of memory. very
> >> close
> >> >>> to
> >> >>> the setting of -Xmx6G limit.
> >> >>> and very soon the regionserver will be OOME.
> >> >>>
> >> >>>
> >> >>> if the above holds true, that means a regionserver can only support
> >> >>> around
> >> >>> 60 regions. I configured the region size to be default 256M.
> >> >>>
> >> >>> I checked the load on the server and the load is very light. cpu and
> >> disk
> >> >>> usage is very low. What can I do to drastically increase the number
> of
> >> >>> regions
> >> >>> supported per regionserver in hbase ?
> >> >>>
> >> >>>
> >> >>> Jimmy
> >> >>>
> >> >>
> >> >
> >>
> >
>
