[
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-19937:
-----------------------------------
Fix Version/s: (was: 1.5.0)
> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer
> are initialized
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
> Affects Versions: 2.0.0-beta-2
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Fix For: 2.0.0-beta-2, 1.4.2, 2.0.0
>
> Attachments: HBASE-19937.branch-2.001.patch,
> HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch,
> HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch,
> HBASE-19937.branch-2.006.patch, import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master
> load system coprocessors in
>
> {code:java}
> 844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it
> will encounter Exception as follows,
>
> {noformat}
> Exception in thread
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
> java.lang.IllegalArgumentException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor initialized by calling
> {code:java}
> 848 startServiceThreads();{code}
> And LoadBalancer is initialized by calling
> {code:java}
> 868 this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS;
> CreateTableProcedure table=hbase:rsgroup
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>
> As a result of CreateTableProcedure.rollbackState, it may then print logs
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> Failed to perform check
> java.io.IOException: Failed to create group table.
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>
> After some auto-retries, it loops running the thread RSGroupStartupWorker,
> will print logs as follows,
> 2018-02-02,16:23:17,626 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,730 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,834 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,937 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
>
> And using shells of rsgroup, it will tips that currently is in "offline mode".
>
> The reason of this problem is that the order of createRSGroupTable and
> initializing of ProcedureExecutor and LoadBalancer is out of control. If the
> former is excuted earlier, it will encounter Exception mentioned before.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
