[
https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356833#comment-16356833
]
Xiaolin Ha edited comment on HBASE-19937 at 2/8/18 3:03 PM:
------------------------------------------------------------
The patch 002 is with a UT which can make the bug appear. Please ensure the
createRSGroupTable is called in a run.
You can notice the log:
{code:java}
2018-02-08 19:16:47,398 INFO
[org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130]
master.HMaster(1791): Client=null/null create 'hbase:rsgroup',
{TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|',
METADATA => {'SPLIT_POLICY' =>
'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME =>
'm', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR =>
'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
'true', BLOCKSIZE => '65536'}
{code}
The patch 001 has fixed this problem.
was (Author: xiaolin ha):
The patch 002 is add a UT which can make the bug appear. Please ensure the
createRSGroupTable is called in a run.
You can notice the log:
{code:java}
2018-02-08 19:16:47,398 INFO
[org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130]
master.HMaster(1791): Client=null/null create 'hbase:rsgroup',
{TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|',
METADATA => {'SPLIT_POLICY' =>
'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME =>
'm', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR =>
'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
'true', BLOCKSIZE => '65536'}
{code}
The patch 001 has fixed this problem.
> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer
> are initialized
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
> Affects Versions: 2.0.0-beta-2
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: HBASE-19937.branch-2.001.patch,
> HBASE-19937.branch-2.002.patch
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master
> load system coprocessors in
>
> {code:java}
> 844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it
> will encounter Exception as follows,
>
> {noformat}
> Exception in thread
> "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130"
> java.lang.IllegalArgumentException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor initialized by calling
> {code:java}
> 848 startServiceThreads();{code}
> And LoadBalancer is initialized by calling
> {code:java}
> 868 this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will
> encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
> runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS;
> CreateTableProcedure table=hbase:rsgroup
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>
> As a result of CreateTableProcedure.rollbackState, it may then print logs
> warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> Failed to perform check
> java.io.IOException: Failed to create group table.
> org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>
> After some auto-retries, it loops running the thread RSGroupStartupWorker,
> will print logs as follows,
> 2018-02-02,16:23:17,626 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,730 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,834 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
> 2018-02-02,16:23:17,937 INFO
> org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker:
> RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
> rootMetaFound=true
>
> And using shells of rsgroup, it will tips that currently is in "offline mode".
>
> The reason of this problem is that the order of createRSGroupTable and
> initializing of ProcedureExecutor and LoadBalancer is out of control. If the
> former is excuted earlier, it will encounter Exception mentioned before.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
