If Andrew agrees, I can do the backport. Cheers
On Fri, Oct 30, 2015 at 7:29 PM, Heng Chen <[email protected]> wrote: > Any plan backport HBASE-14309 into 0.98 ? > > 2015-10-31 10:26 GMT+08:00 Heng Chen <[email protected]>: > > > bq. Not running balancer because 3 region(s) in transition > > > > Yeah, but balancer runs every 5 minutes, 3 region(s) in transition log > > only show up only in a few places.... > > > > 2015-10-31 10:23 GMT+08:00 Ted Yu <[email protected]>: > > > >> bq. there are no logs to record why we not running balancer > >> > >> Here was the reason: > >> > >> bq. Not running balancer because 3 region(s) in transition: > >> > >> bq. Could we just balance regions not in transition? > >> > >> Yes. Please take a look at HBASE-14309 > >> > >> Cheers > >> > >> On Fri, Oct 30, 2015 at 7:19 PM, Heng Chen <[email protected]> > >> wrote: > >> > >> > My hbase cluster version is 0.98.6 > >> > > >> > There are lots of regions on it, about 10000+ > >> > > >> > Load is heavy, almost every time there are regions in split.... > >> > > >> > So i found that the balancer not run for a long time. > >> > > >> > grep -i 'balancer' master.log, there are only logs like below > >> > > >> > 2015-09-30 11:29:13,994 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {30971a1ae707b9f5bbcd7b8802f32059={30971a1ae707b9f5bbcd7b8802f32059 > >> > state=SPLITTING_NEW, ts=1443583753692, > >> > server=dx-ape-regionserver30-online,60020,1440183710528}, > >> > 13eaacf6df912d0cb598067610c5a85f={13eaacf6df912d0cb598067610c5a85f > >> > state=SPLITTING_NEW, ... > >> > 2015-10-01 17:44:14,032 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {55fc1c408832233ee1dd01c70c61ae14={55fc1c408832233ee1dd01c70c61ae14 > >> > state=SPLITTING, ts=1443692653425, > >> > server=dx-ape-regionserver27-online,60020,1440183264316}, > >> > 07439db0ff1319d20b43aa4d2e43a4ae={07439db0ff1319d20b43aa4d2e43a4ae > >> > state=SPLITTING_NEW, ts=1... > >> > 2015-10-04 14:04:14,126 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {2bd0891dc9ca5fb15ea8b661127193b7={2bd0891dc9ca5fb15ea8b661127193b7 > >> > state=SPLITTING, ts=1443938653837, > >> > server=dx-ape-regionserver9-online,60020,1440182448264}, > >> > 76bbb47201c3958e3a9c1086bfb351c5={76bbb47201c3958e3a9c1086bfb351c5 > >> > state=SPLITTING_NEW, ts=14... > >> > 2015-10-05 14:14:14,161 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {669719254f132476c6df0e0e9b1fc93f={669719254f132476c6df0e0e9b1fc93f > >> > state=SPLITTING_NEW, ts=1444025653911, > >> > server=dx-ape-regionserver1-online,60020,1440178926883}, > >> > ec612addaabb22c8f46b2c903bd1158b={ec612addaabb22c8f46b2c903bd1158b > >> > state=SPLITTING_NEW, t... > >> > 2015-10-15 21:19:14,512 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {2b7a5c3ddc7ee919199c68611e6f6c96={2b7a5c3ddc7ee919199c68611e6f6c96 > >> > state=SPLITTING, ts=1444915153714, > >> > server=dx-ape-regionserver12-online,60020,1440181883146}, > >> > cda06b9ebd651c616361f73a469a1a52={cda06b9ebd651c616361f73a469a1a52 > >> > state=SPLITTING_NEW, ts=1... > >> > 2015-10-15 23:39:14,513 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {b1d3429606407280e442d8ce3de873c4={b1d3429606407280e442d8ce3de873c4 > >> > state=SPLITTING, ts=1444923553844, > >> > server=dx-ape-regionserver25-online,60020,1440183200463}, > >> > ae7ba7ee139c7ba84ba707671b7959c4={ae7ba7ee139c7ba84ba707671b7959c4 > >> > state=SPLITTING_NEW, ts=1... > >> > 2015-10-21 19:29:14,692 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {e677e41a383eb20429c9906bafc252bb={e677e41a383eb20429c9906bafc252bb > >> > state=SPLITTING_NEW, ts=1445426954437, > >> > server=dx-ape-regionserver11-online,60020,1440181972615}, > >> > 0028b035271bdd6d30e7fb6f1ffb406d={0028b035271bdd6d30e7fb6f1ffb406d > >> > state=SPLITTING, ts=1... > >> > 2015-10-25 10:24:14,790 DEBUG > >> > [dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] > >> master.HMaster: > >> > Not running balancer because 3 region(s) in transition: > >> > {694912c058fcd0e6bff7b3eaed1b051b={694912c058fcd0e6bff7b3eaed1b051b > >> > state=SPLITTING_NEW, ts=1445739851757, > >> > server=dx-ape-regionserver27-online,60020,1440183264316}, > >> > 7859193f7ca5ee2c98636cb812b549a7={7859193f7ca5ee2c98636cb812b549a7 > >> > state=SPLITTING, ts=1... > >> > > >> > > >> > The balancer runs every 5 minutes, there are no logs to record why we > >> not > >> > running balancer, should we add some logs at least? > >> > > >> > As for the above logs, it seems we stop running balancer when regions > >> in > >> > transition > >> > > >> > This is the relates code > >> > > >> > // Only allow one balance run at at time. > >> > if (this.assignmentManager.getRegionStates().isRegionsInTransition()) > { > >> > Map<String, RegionState> regionsInTransition = > >> > this.assignmentManager.getRegionStates().getRegionsInTransition(); > >> > LOG.debug("Not running balancer because " + > >> regionsInTransition.size() + > >> > " region(s) in transition: " + > org.apache.commons.lang.StringUtils. > >> > abbreviate(regionsInTransition.toString(), 256)); > >> > return false; > >> > } > >> > > >> > And i have questions, why we use regions states to avoid more than > >> > one balancer running? > >> > > >> > Could we just balance regions not in transition? > >> > > >> > > >> > Thanks! > >> > > >> > > > > >
