[
https://issues.apache.org/jira/browse/HBASE-20690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784549#comment-16784549
]
Xiang Li edited comment on HBASE-20690 at 3/5/19 4:18 PM:
----------------------------------------------------------
Hi [~xucang] I have been working on this JIRA for some days, and would like to
provide some updates.
# I think the patch v002 uploaded by [~andrewcheng] is the right way to go
# Some potential issues are also introduced by patch v002, as described below.
Patch v002 moves the following operations from postCreateTable() to
preCreateTableAction()
# Decide which rsgroup to go to according to table's namespace
# Call RSGroupInfoManagerImpl#moveTables() to update the rsgroup information
(but do not move the regions actually)
The changes above affect the procedure to create hbase:rsgroup table during
HMaster starts, because preCreateTableAction() is a part of
HMaster#createSystemTable(), while postCreateTable() is not.
*Issue 1: Trying to read cluster schema service when it is not ready during
HMaster starts*
when HMaster starts, a race condition could be triggered on cluster schema
service. In HMaster#finishActiveMasterInitialization(), hbase:rsgroup is
created by
{code:title=HMaster#finishActiveMasterInitialization()}
this.balancer.initialize(); // line 1060
{code}
by HMaster#createSystemTable() internally, in which, a CreateTableProcedure is
scheduled. preCreateTableAction() is called and the following statement is
called to determine rsgroup according to table's namespace:
{code}
String groupName =
master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
{code}
But getClusterSchema might return null because the cluster schema service is
not ready yet. Actually, it is not ready until the following statement is
called as
{code:title=HMaster#finishActiveMasterInitialization()}
initClusterSchemaService(); // line 1132
{code}
As for the solution, I added a loop of sleep and do not try to getClusterSchema
until it is not null.
But I notices that there are several places in which
master.getClusterSchema.get...() is called without NPE check or protection.
*Issue 2: RSGroupInfoManagerImpl.RSGroupStartupWorker#createRSGroupTable()
throws IOException with "Only default servers can be updated during offline
mode"*
In RSGroupInfoManagerImpl.RSGroupStartupWorker#waitForGroupTableOnline(),
createRSGroupTable() is called (line 771 of RSGroupInfoManagerImpl.java) before
online is set to true (line 780). When offline mode(online = false),
RSGroupInfoManagerImpl#flushConfig() will throw the exception mentioned above
as we are to add hbase:rsgroup table into default rsgroup which makes the
tables of default rsgroup change.
{code:title=RSGroupInfoManagerImpl}
private synchronized void flushConfig(Map<String, RSGroupInfo> newGroupMap)
throws IOException {
...
if (!m.equals(newGroupMap) ||
!oldDefaultGroup.getTables().equals(newDefaultGroup.getTables())) { // here
throw new IOException("Only default servers can be updated during
offline mode");
{code}
This is not a race condition. And originally, without the patch, flushConfig()
is not a part of preCreateTableAction(), so not a part of
HMaster#createSystemTable(), so it is not called when offline mode during
HMaster starts.
As for the solution, I have no idea yet. I am not sure if we could skip the
check above when HMaster's initialization is not finished yet
was (Author: water):
Hi [~xucang] I have been working on this JIRA for some days, and would like to
provide some updates.
# I think the patch v002 uploaded by [~andrewcheng] is the right way to go
# Some potential issues are also introduced by patch v002, as described below.
Patch v002 moves the following operations from postCreateTable() to
preCreateTableAction()
# Decide which rsgroup to go to according to table's namespace
# Call RSGroupInfoManagerImpl#moveTables() to update the rsgroup information
(but do not move the regions actually)
The changes above affect the procedure to create hbase:rsgroup table during
HMaster starts, because preCreateTableAction() is a part of
HMaster#createSystemTable(), while postCreateTable() is not.
*Issue 1: Trying to read cluster schema service when it is not ready during
HMaster starts*
when HMaster starts, a race condition could be triggered on cluster schema
service. In HMaster#finishActiveMasterInitialization(), hbase:rsgroup is
created by
{code:title=HMaster#finishActiveMasterInitialization()}
this.balancer.initialize(); // line 1060
{code}
by HMaster#createSystemTable() internally, in which, a CreateTableProcedure is
scheduled. preCreateTableAction() is called and the following statement is
called to determine rsgroup according to table's namespace:
{code}
String groupName =
master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
{code}
But getClusterSchema might return null because the cluster schema service is
not ready yet. Actually, it is not ready until the following statement is
called as
{code:title=HMaster#finishActiveMasterInitialization()}
initClusterSchemaService(); // line 1132
{code}
As for the solution, I added a loop of sleep and do not try to getClusterSchema
until it is not null.
But I notices that there are several places in which
master.getClusterSchema.get...() is called without NPE check or protection.
*Issue 2: RSGroupInfoManagerImpl.RSGroupStartupWorker#createRSGroupTable()
throws IOException with "Only default servers can be updated during offline
mode"*
In RSGroupInfoManagerImpl.RSGroupStartupWorker#waitForGroupTableOnline(),
createRSGroupTable() is called (line 771 of RSGroupInfoManagerImpl.java) before
online is set to true (line 780). When offline mode(online = false),
RSGroupInfoManagerImpl#flushConfig() will throw the exception mentioned above
as we are to add hbase:rsgroup table into default rsgroup which makes the
tables of default rsgroup change.
{code:title=RSGroupInfoManagerImpl}
private synchronized void flushConfig(Map<String, RSGroupInfo> newGroupMap)
throws IOException {
...
if (!m.equals(newGroupMap) ||
!oldDefaultGroup.getTables().equals(newDefaultGroup.getTables())) { // here
throw new IOException("Only default servers can be updated during
offline mode");
{code}
This is not a race condition. And originally, without the patch, flushConfig()
is not a part of preCreateTableAction(), so not a part of
HMaster#createSystemTable(), so it is not called when offline mode during
HMaster starts.
As for the solution, I have no idea yet. I am not sure if we could skip the
check above for system table.
> Moving table to target rsgroup needs to handle TableStateNotFoundException
> --------------------------------------------------------------------------
>
> Key: HBASE-20690
> URL: https://issues.apache.org/jira/browse/HBASE-20690
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Xiang Li
> Priority: Major
> Attachments: HBASE-20690.master.001.patch,
> HBASE-20690.master.002.patch
>
>
> This is related code:
> {code}
> if (targetGroup != null) {
> for (TableName table: tables) {
> if (master.getAssignmentManager().isTableDisabled(table)) {
> LOG.debug("Skipping move regions because the table" + table + " is
> disabled.");
> continue;
> }
> {code}
> In a stack trace [~rmani] showed me:
> {code}
> 2018-06-06 07:10:44,893 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=20000]
> master.TableStateManager: Unable to get table demo:tbl1 state
> org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException:
> demo:tbl1
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:193)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:143)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.isTableDisabled(AssignmentManager.java:346)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveTables(RSGroupAdminServer.java:407)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.assignTableToGroup(RSGroupAdminEndpoint.java:447)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.postCreateTable(RSGroupAdminEndpoint.java:470)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$12.call(MasterCoprocessorHost.java:334)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$12.call(MasterCoprocessorHost.java:331)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:331)
> at org.apache.hadoop.hbase.master.HMaster$3.run(HMaster.java:1768)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1750)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:593)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logic should take potential TableStateNotFoundException into account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
