[
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-19937:
-------------------------------
Description:
rsgroup table will be created by calling createRSGroupTable when master load
system coprocessors in
{code:java}
844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
when ProcedureExecutor hasn't been initialized before createRSGroupTable, it
will encounters Exception as follows,
{noformat}
Exception in thread
"org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
java.lang.IllegalArgumentException
at
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
at
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
at
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
at
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
while ProcedureExecutor initialized by calling
{code:java}
848 startServiceThreads();{code}
And LoadBalancer is initialized by calling
{code:java}
868 this.balancer.initialize();{code}
When LoadBalancer hasn't been initialized before createRSGroupTable, it will
encounters Exception as follows,
2018-02-02,16:12:45,688 ERROR
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS;
CreateTableProcedure table=hbase:rsgroup
java.lang.NullPointerException
at
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
at
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
As a result of CreateTableProcedure.rollbackState, it may then print logs
warning TableExistsException as follows,
2018-02-02,16:12:55,503 WARN
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
Failed to perform check
java.io.IOException: Failed to create group table.
org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
at
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
After some auto-retries, it loops running the thread RSGroupStartupWorker, will
print logs as follows,
2018-02-02,16:23:17,626 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
2018-02-02,16:23:17,730 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
2018-02-02,16:23:17,834 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
2018-02-02,16:23:17,937 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
And using shells of rsgroup, it will tips that currently is in "offline mode".
The reason of this problem is that the order of createRSGroupTable and
initializing of ProcedureExecutor and LoadBalancer is out of control. If the
former is excuted earlier, it will encounter Exception mentioned before.
was:
When enable rsgroup, it may throws NPE as follows,
2018-02-02,16:12:45,688 ERROR
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS;
CreateTableProcedure table=hbase:rsgroup
java.lang.NullPointerException
at
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
at
org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
As a result of CreateTableProcedure.rollbackState, it may then print logs
warning TableExistsException as follows,
2018-02-02,16:12:55,503 WARN
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
Failed to perform check
java.io.IOException: Failed to create group table.
org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
at
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
After some auto-retries, it loops running the thread RSGroupStartupWorker, will
print logs as follows,
2018-02-02,16:23:17,626 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
2018-02-02,16:23:17,730 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
2018-02-02,16:23:17,834 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
2018-02-02,16:23:17,937 INFO
org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
rootMetaFound=true
And using shells of rsgroup, it will tips that currently is in "offline mode".
The reason of this problem is that CreateTableProcedure used
RSGroupBasedLoadBalancer, who has member variables initialized depending on
return of CreateTableProcedure.
Summary: createRSGroupTable may encounter Exceptions before
ProcedureExecutor or LoadBalancer is initialized (was: Enable rsgroup NPE in
CreateTableProcedure)
> createRSGroupTable may encounter Exceptions before ProcedureExecutor or
> LoadBalancer is initialized
> ----------------------------------------------------------------------------------------------------
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
> Affects Versions: 2.0.0-beta-2
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch
>
>
> rsgroup table will be created by calling createRSGroupTable when master load
> system coprocessors in
>
> {code:java}
> 844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it
> will encounters Exception as follows,
>
> {noformat}
> Exception in thread
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
> java.lang.IllegalArgumentException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor initialized by calling
> {code:java}
> 848 startServiceThreads();{code}
> And LoadBalancer is initialized by calling
> {code:java}
> 868 this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS;
> CreateTableProcedure table=hbase:rsgroup
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>
> As a result of CreateTableProcedure.rollbackState, it may then print logs
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> Failed to perform check
> java.io.IOException: Failed to create group table.
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>
> After some auto-retries, it loops running the thread RSGroupStartupWorker,
> will print logs as follows,
> 2018-02-02,16:23:17,626 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,730 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,834 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,937 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
>
> And using shells of rsgroup, it will tips that currently is in "offline mode".
>
> The reason of this problem is that the order of createRSGroupTable and
> initializing of ProcedureExecutor and LoadBalancer is out of control. If the
> former is excuted earlier, it will encounter Exception mentioned before.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
