My hbase cluster version is 0.98.6
There are lots of regions on it, about 10000+
Load is heavy, almost every time there are regions in split....
So i found that the balancer not run for a long time.
grep -i 'balancer' master.log, there are only logs like below
2015-09-30 11:29:13,994 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{30971a1ae707b9f5bbcd7b8802f32059={30971a1ae707b9f5bbcd7b8802f32059
state=SPLITTING_NEW, ts=1443583753692,
server=dx-ape-regionserver30-online,60020,1440183710528},
13eaacf6df912d0cb598067610c5a85f={13eaacf6df912d0cb598067610c5a85f
state=SPLITTING_NEW, ...
2015-10-01 17:44:14,032 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{55fc1c408832233ee1dd01c70c61ae14={55fc1c408832233ee1dd01c70c61ae14
state=SPLITTING, ts=1443692653425,
server=dx-ape-regionserver27-online,60020,1440183264316},
07439db0ff1319d20b43aa4d2e43a4ae={07439db0ff1319d20b43aa4d2e43a4ae
state=SPLITTING_NEW, ts=1...
2015-10-04 14:04:14,126 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{2bd0891dc9ca5fb15ea8b661127193b7={2bd0891dc9ca5fb15ea8b661127193b7
state=SPLITTING, ts=1443938653837,
server=dx-ape-regionserver9-online,60020,1440182448264},
76bbb47201c3958e3a9c1086bfb351c5={76bbb47201c3958e3a9c1086bfb351c5
state=SPLITTING_NEW, ts=14...
2015-10-05 14:14:14,161 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{669719254f132476c6df0e0e9b1fc93f={669719254f132476c6df0e0e9b1fc93f
state=SPLITTING_NEW, ts=1444025653911,
server=dx-ape-regionserver1-online,60020,1440178926883},
ec612addaabb22c8f46b2c903bd1158b={ec612addaabb22c8f46b2c903bd1158b
state=SPLITTING_NEW, t...
2015-10-15 21:19:14,512 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{2b7a5c3ddc7ee919199c68611e6f6c96={2b7a5c3ddc7ee919199c68611e6f6c96
state=SPLITTING, ts=1444915153714,
server=dx-ape-regionserver12-online,60020,1440181883146},
cda06b9ebd651c616361f73a469a1a52={cda06b9ebd651c616361f73a469a1a52
state=SPLITTING_NEW, ts=1...
2015-10-15 23:39:14,513 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{b1d3429606407280e442d8ce3de873c4={b1d3429606407280e442d8ce3de873c4
state=SPLITTING, ts=1444923553844,
server=dx-ape-regionserver25-online,60020,1440183200463},
ae7ba7ee139c7ba84ba707671b7959c4={ae7ba7ee139c7ba84ba707671b7959c4
state=SPLITTING_NEW, ts=1...
2015-10-21 19:29:14,692 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{e677e41a383eb20429c9906bafc252bb={e677e41a383eb20429c9906bafc252bb
state=SPLITTING_NEW, ts=1445426954437,
server=dx-ape-regionserver11-online,60020,1440181972615},
0028b035271bdd6d30e7fb6f1ffb406d={0028b035271bdd6d30e7fb6f1ffb406d
state=SPLITTING, ts=1...
2015-10-25 10:24:14,790 DEBUG
[dx-ape-hmaster1-online,60000,1438752227040-BalancerChore] master.HMaster:
Not running balancer because 3 region(s) in transition:
{694912c058fcd0e6bff7b3eaed1b051b={694912c058fcd0e6bff7b3eaed1b051b
state=SPLITTING_NEW, ts=1445739851757,
server=dx-ape-regionserver27-online,60020,1440183264316},
7859193f7ca5ee2c98636cb812b549a7={7859193f7ca5ee2c98636cb812b549a7
state=SPLITTING, ts=1...
The balancer runs every 5 minutes, there are no logs to record why we not
running balancer, should we add some logs at least?
As for the above logs, it seems we stop running balancer when regions in
transition
This is the relates code
// Only allow one balance run at at time.
if (this.assignmentManager.getRegionStates().isRegionsInTransition()) {
Map<String, RegionState> regionsInTransition =
this.assignmentManager.getRegionStates().getRegionsInTransition();
LOG.debug("Not running balancer because " + regionsInTransition.size() +
" region(s) in transition: " + org.apache.commons.lang.StringUtils.
abbreviate(regionsInTransition.toString(), 256));
return false;
}
And i have questions, why we use regions states to avoid more than
one balancer running?
Could we just balance regions not in transition?
Thanks!
