bq. there are no logs to record why we not running balancer Here was the reason:
bq. Not running balancer because 3 region(s) in transition: bq. Could we just balance regions not in transition? Yes. Please take a look at HBASE-14309 Cheers On Fri, Oct 30, 2015 at 7:19 PM, Heng Chen <[email protected]> wrote: > My hbase cluster version is 0.98.6 > > There are lots of regions on it, about 10000+ > > Load is heavy, almost every time there are regions in split.... > > So i found that the balancer not run for a long time. > > grep -i 'balancer' master.log, there are only logs like below > > 2015-09-30 11:29:13,994 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {30971a1ae707b9f5bbcd7b8802f32059={30971a1ae707b9f5bbcd7b8802f32059 > state=SPLITTING_NEW, ts=1443583753692, > server=dx-ape-regionserver30-online,60020,1440183710528}, > 13eaacf6df912d0cb598067610c5a85f={13eaacf6df912d0cb598067610c5a85f > state=SPLITTING_NEW, ... > 2015-10-01 17:44:14,032 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {55fc1c408832233ee1dd01c70c61ae14={55fc1c408832233ee1dd01c70c61ae14 > state=SPLITTING, ts=1443692653425, > server=dx-ape-regionserver27-online,60020,1440183264316}, > 07439db0ff1319d20b43aa4d2e43a4ae={07439db0ff1319d20b43aa4d2e43a4ae > state=SPLITTING_NEW, ts=1... > 2015-10-04 14:04:14,126 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {2bd0891dc9ca5fb15ea8b661127193b7={2bd0891dc9ca5fb15ea8b661127193b7 > state=SPLITTING, ts=1443938653837, > server=dx-ape-regionserver9-online,60020,1440182448264}, > 76bbb47201c3958e3a9c1086bfb351c5={76bbb47201c3958e3a9c1086bfb351c5 > state=SPLITTING_NEW, ts=14... > 2015-10-05 14:14:14,161 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {669719254f132476c6df0e0e9b1fc93f={669719254f132476c6df0e0e9b1fc93f > state=SPLITTING_NEW, ts=1444025653911, > server=dx-ape-regionserver1-online,60020,1440178926883}, > ec612addaabb22c8f46b2c903bd1158b={ec612addaabb22c8f46b2c903bd1158b > state=SPLITTING_NEW, t... > 2015-10-15 21:19:14,512 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {2b7a5c3ddc7ee919199c68611e6f6c96={2b7a5c3ddc7ee919199c68611e6f6c96 > state=SPLITTING, ts=1444915153714, > server=dx-ape-regionserver12-online,60020,1440181883146}, > cda06b9ebd651c616361f73a469a1a52={cda06b9ebd651c616361f73a469a1a52 > state=SPLITTING_NEW, ts=1... > 2015-10-15 23:39:14,513 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {b1d3429606407280e442d8ce3de873c4={b1d3429606407280e442d8ce3de873c4 > state=SPLITTING, ts=1444923553844, > server=dx-ape-regionserver25-online,60020,1440183200463}, > ae7ba7ee139c7ba84ba707671b7959c4={ae7ba7ee139c7ba84ba707671b7959c4 > state=SPLITTING_NEW, ts=1... > 2015-10-21 19:29:14,692 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {e677e41a383eb20429c9906bafc252bb={e677e41a383eb20429c9906bafc252bb > state=SPLITTING_NEW, ts=1445426954437, > server=dx-ape-regionserver11-online,60020,1440181972615}, > 0028b035271bdd6d30e7fb6f1ffb406d={0028b035271bdd6d30e7fb6f1ffb406d > state=SPLITTING, ts=1... > 2015-10-25 10:24:14,790 DEBUG > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster: > Not running balancer because 3 region(s) in transition: > {694912c058fcd0e6bff7b3eaed1b051b={694912c058fcd0e6bff7b3eaed1b051b > state=SPLITTING_NEW, ts=1445739851757, > server=dx-ape-regionserver27-online,60020,1440183264316}, > 7859193f7ca5ee2c98636cb812b549a7={7859193f7ca5ee2c98636cb812b549a7 > state=SPLITTING, ts=1... > > > The balancer runs every 5 minutes, there are no logs to record why we not > running balancer, should we add some logs at least? > > As for the above logs, it seems we stop running balancer when regions in > transition > > This is the relates code > > // Only allow one balance run at at time. > if (this.assignmentManager.getRegionStates().isRegionsInTransition()) { > Map<String, RegionState> regionsInTransition = > this.assignmentManager.getRegionStates().getRegionsInTransition(); > LOG.debug("Not running balancer because " + regionsInTransition.size() + > " region(s) in transition: " + org.apache.commons.lang.StringUtils. > abbreviate(regionsInTransition.toString(), 256)); > return false; > } > > And i have questions, why we use regions states to avoid more than > one balancer running? > > Could we just balance regions not in transition? > > > Thanks! >
