When triggerring full GC, better do in rolling fashion so that region servers don't incur pause at the same time.
FYI On Wed, Oct 7, 2015 at 8:24 PM, Vladimir Rodionov <[email protected]> wrote: > Can you trigger full GC across the cluster? If slow queries will go down > after that - you get the answer. > GC s-t-w frequency and duration may increase steadily until full reset > (full GC) is executed, I think > > -Vlad > > On Wed, Oct 7, 2015 at 6:35 PM, UI-JIN LIM <[email protected]> wrote: > > > I have post images to hosting service and share again!! > > > > --- > > Hi. This is Ui Jin, Lim in Korea, LG CNS > > > > We had setup and are operating hbase 0.98.13 on our customer, iptv > Company. > > Our customer claim about this spike. Following is our status. > > > > . We had been operating with BlockCache(heap) for 3 months. The ratio of > > above 100ms has been reached 6.69% of total transaction. > > The following graph show the trend of executions, above 100ms per minute, > > at June 17th. (This is the response time of 2~3 Hbase transactions) > > <Click to see Graph> > > http://i57.tinypic.com/mwcvtj.jpg > > > > - After Hbase restarting at June 18th, The spikes had been reduced > > dramatically. But its ratio had been increasing continuously. > > <Click to see Graph> > > http://i57.tinypic.com/71pfee.jpg > > > > - We had adopted off-heap bucket cache at August 19th. But the increasing > > does not disappear. > > <Click to see Graph> > > http://i61.tinypic.com/2aezazt.jpg > > http://i58.tinypic.com/svm32x.jpg > > > > - We set HBase to log slow query which have above 10ms execution time. > All > > RegionServer logs show increasing its counts. > > <Click to see Graph> > > http://i59.tinypic.com/68u1ci.jpg > > > > What we concern is not the spike itself but its incremental trend. > > There is no increment of transactions and dataset. > > I ask your thoughts about this issue. > > > > > > -----Original Message----- > > From: Vladimir Rodionov [mailto:[email protected]] > > Sent: Thursday, October 08, 2015 9:50 AM > > To: [email protected] > > Subject: Re: Asking about the trend of increasing latency, hbase spikes. > > > > That is definitely, interesting graphs, but we do not see them. > > Can you host them somewhere and post here the links? > > > > -Vlad > > > > On Wed, Oct 7, 2015 at 5:38 PM, Biju N <[email protected]> wrote: > > > > > Looks like the attachments are not viewable. > > > > > > On Wed, Oct 7, 2015 at 8:35 PM, UI-JIN LIM <[email protected]> wrote: > > > > > > > Hi. This is Ui Jin, Lim in Korea, LG CNS > > > > > > > > > > > > > > > > We had setup and are operating hbase 0.98.13 on our customer, iptv > > > > Company. > > > > > > > > Our customer claim about this spike. Following is our status. > > > > > > > > > > > > > > > > . We had been operating with BlockCache(heap) for 3 months. The > > > > ratio of above 100ms has been reached 6.69% of total transaction. > > > > > > > > The following graph show the trend of executions, above 100ms per > > > > minute, at June 17th. (This is the response time of 2~3 Hbase > > > > transactions) > > > > > > > > [image: cid:[email protected]] > > > > > > > > > > > > > > > > - After Hbase restarting at June 18th, The spikes had been reduced > > > > dramatically. But its ratio had been increasing continuously. > > > > > > > > [image: cid:[email protected]] > > > > > > > > > > > > > > > > - We had adopted off-heap bucket cache at August 19th. But the > > > > increasing does not disappear. > > > > > > > > [image: cid:[email protected]] > > > > > > > > [image: cid:[email protected]] > > > > > > > > > > > > > > > > - We set HBase to log slow query which have above 10ms execution > time. > > > All > > > > RegionServer logs show increasing its counts. > > > > > > > > > > > > > > > > [image: cid:[email protected]] > > > > > > > > > > > > > > > > > > > > > > > > What we concern is not the spike itself but its incremental trend. > > > > > > > > There is no increment of transactions and dataset. > > > > > > > > I ask your thoughts about this issue. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------- > > > > > > > > *UI JIN, LIM* > > > > > > > > *DNA 3.0** - **LG CNS* > > > > > > > > Data Architecture Advisory > > > > > > > > Big Data Technical Support Team > > > > > > > > Mobile : +82) 010-8307-1210 > > > > > > > > > > > > > > > > > >
