[ 
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-19937:
-------------------------------
    Summary: Ensure createRSGroupTable be called after ProcedureExecutor and 
LoadBalancer are initialized   (was: createRSGroupTable may encounter 
Exceptions before ProcedureExecutor or LoadBalancer is initialized )

> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer 
> are initialized 
> ---------------------------------------------------------------------------------------------
>
>                 Key: HBASE-19937
>                 URL: https://issues.apache.org/jira/browse/HBASE-19937
>             Project: HBase
>          Issue Type: Bug
>          Components: rsgroup
>    Affects Versions: 2.0.0-beta-2
>            Reporter: Xiaolin Ha
>            Assignee: Xiaolin Ha
>            Priority: Major
>         Attachments: HBASE-19937.branch-2.001.patch
>
>
> rsgroup table will be created by calling createRSGroupTable when master load 
> system coprocessors in 
>  
> {code:java}
> 844  this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it 
> will encounters Exception as follows,
>  
> {noformat}
> Exception in thread 
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
>  java.lang.IllegalArgumentException
> at 
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor  initialized by calling
> {code:java}
> 848  startServiceThreads();{code}
> And LoadBalancer is initialized by calling 
> {code:java}
> 868  this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will 
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught 
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; 
> CreateTableProcedure table=hbase:rsgroup
>  java.lang.NullPointerException
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
>  at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
>  at 
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
>  at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
>  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
>  at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>  
> As a result of CreateTableProcedure.rollbackState, it may then print logs 
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> Failed to perform check
>  java.io.IOException: Failed to create group table. 
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
>  at 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>  
> After some auto-retries, it loops running the thread RSGroupStartupWorker, 
> will print logs as follows, 
> 2018-02-02,16:23:17,626 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,730 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,834 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  2018-02-02,16:23:17,937 INFO 
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: 
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, 
> rootMetaFound=true
>  
> And using shells of rsgroup, it will tips that currently is in "offline mode".
>  
> The reason of this problem is that the order of createRSGroupTable and 
> initializing of ProcedureExecutor and LoadBalancer is out of control. If the 
> former is excuted earlier, it will encounter Exception mentioned before.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to