Please try to determine how much percent of the reqs/sec metric is due to read.
At this level, do your queries / jobs get slower ? Cheers On Sun, Jan 25, 2015 at 6:48 PM, Pun Intended <[email protected]> wrote: > Sorry for the confusion. I should have used the same place to grab the > numbers from. > > In my original email, I grabbed the reqs/sec metric from the Hbase UI > directly. It was showing 1000 Reqs/sec. Right now, when I look at that, the > number is 891 Reqs/sec, but it fluctuates. > > The 55K Reqs/sec in my last email, I got from Cloudera Manager. I am not > sure why it is reporting a different number. It was 55K Reqs/sec, but now > it jumped back up to 60K Reqs/sec. > > > > On Sun, Jan 25, 2015 at 9:37 PM, Ted Yu <[email protected]> wrote: > > > From your first email: > > > > bq. 1000reqs/sec vs 4-5 reqs/sec elsewhere > > > > Now the request count becomes 55K requests/sec ? > > Just want to clarify whether the request count increased or decreased > when > > compaction is not in progress. > > > > Cheers > > > > On Sun, Jan 25, 2015 at 6:23 PM, Pun Intended <[email protected]> > > wrote: > > > > > Thanks! > > > I ran hbase hbck and all seems healthy. > > > > > > 0 inconsistencies detected. > > > Status: OK > > > > > > Also the meta region shows to be 100% compacted, but I still see 55K > > > requests/sec to that RegionServer: > > > > > > Region > > > Name Num. Compacting KVs Num. > > > Compacted KVs Compaction Progress > > > > > > hbase:meta,,1.1588230740 > > > 195349 195349 100.00% > > > > > > best > > > > > > > > > On Sun, Jan 25, 2015 at 8:28 PM, Ted Yu <[email protected]> wrote: > > > > > > > After the adjustment, hbase:meta would still be accessed for region > > > > locations. But the request count should come down noticeably. > > > > > > > > bq. Is there a way to inspect whether I have corruptions in the META > > > table? > > > > > > > > Please take a look at: > > > > http://hbase.apache.org/book.html#hbck.in.depth > > > > > > > > Cheers > > > > > > > > On Sun, Jan 25, 2015 at 5:20 PM, Pun Intended < > [email protected] > > > > > > > wrote: > > > > > > > > > Will do! Thanks you for your suggestion. I will keep an eye on the > > read > > > > > rate to that RegionServer, but in case it doesn't drop, do you > think > > > > there > > > > > should be something else I need to look at? > > > > > > > > > > * Is there a way to inspect whether I have corruptions in the META > > > table? > > > > > * Any other possible misconfiguration that comes to mind that I > > should > > > > keep > > > > > an eye for? > > > > > > > > > > best > > > > > > > > > > On Sun, Jan 25, 2015 at 8:05 PM, Ted Yu <[email protected]> > wrote: > > > > > > > > > > > Suggest you lower the frequency of compacting hbase:meta > > > > > > > > > > > > Move the second command to another script. > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > > > > > > > > > > On Jan 25, 2015, at 4:46 PM, Pun Intended < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Ted, > > > > > > > > > > > > > > I have a small script: major_compact_hbase.hbsh > > > > > > > > > > > > > > major_compact 'users' > > > > > > > major_compact 'hbase:meta' > > > > > > > exit > > > > > > > > > > > > > > > > > > > > > and a cron calls it once a day with: > > > > > > > /usr/bin/hbase shell major_compact_hbase.hbsh > > > > > > > > > > > > > > Thanks again! > > > > > > > > > > > > > > > > > > > > >> On Sun, Jan 25, 2015 at 7:38 PM, Ted Yu <[email protected]> > > > > wrote: > > > > > > >> > > > > > > >> How do you trigger major compaction manually ? > > > > > > >> > > > > > > >> Cheers > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >>>> On Jan 25, 2015, at 3:40 PM, Pun Intended < > > > > [email protected]> > > > > > > >>> wrote: > > > > > > >>> > > > > > > >>> Thank you so much for your help! > > > > > > >>> You are right, the slow region server hosts hbase:meta!!! I > am > > > > using > > > > > > >> Hbase > > > > > > >>> 0.96.1.1 > > > > > > >>> > > > > > > >>> I have major compactions disabled. I run major compaction > > > manually > > > > > > once a > > > > > > >>> day and when it dpes that besides requesting major compaction > > on > > > my > > > > > > >> tables, > > > > > > >>> it also requests a major_compact 'hbase:meta'. Is that what > > could > > > > be > > > > > > >>> causing the issue? I did a scan on 'hbase:meta' and I see > that > > it > > > > > > >> contains > > > > > > >>> about the same number of regions as it should - I have two > > tables > > > > > with > > > > > > >>> ~3000 regions each. > > > > > > >>> > > > > > > >>> Do you think I have some misconfiguration? Do I need to do > some > > > > > cleanup > > > > > > >> on > > > > > > >>> the meta region? Am I hitting some Hbase bug? Or am I being > > dumb > > > > > > >> requesting > > > > > > >>> major compaction for the meta region daily? > > > > > > >>> > > > > > > >>> I believe on the client side I am doing things right - using > > > > scanner > > > > > > >>> caching, etc. > > > > > > >>> > > > > > > >>> Thank you so much for your help!! > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
