[
https://issues.apache.org/jira/browse/HBASE-13802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564472#comment-14564472
]
Hadoop QA commented on HBASE-13802:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12736067/HBASE-13802.v1.patch
against master branch at commit 325614220f1c208b0846001f903888660096cdb1.
ATTACHMENT ID: 12736067
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1929 checkstyle errors (more than the master's current 1928 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14232//testReport/
Release Findbugs (version 2.0.3) warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14232//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14232//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14232//console
This message is automatically generated.
> Procedure V2: Master fails to come up due to rollback of create namespace
> table
> -------------------------------------------------------------------------------
>
> Key: HBASE-13802
> URL: https://issues.apache.org/jira/browse/HBASE-13802
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
> Affects Versions: 2.0.0, 1.1.0, 1.2.0
> Reporter: Stephen Yuan Jiang
> Assignee: Stephen Yuan Jiang
> Fix For: 2.0.0, 1.2.0, 1.1.1
>
> Attachments: HBASE-13802.v1.patch
>
>
> In Procedure V2 (HBASE-13203) implementation, Rollback of a
> CreateTableProcedure would call the Quota Manager to remove the table from
> namespace quota.
> {code}
> protected static void deleteTableStates(final MasterProcedureEnv env, final
> TableName tableName) {
>
> ProcedureSyncWait.getMasterQuotaManager(env).removeTableFromNamespaceQuota(tableName);
> }
> {code}
> This could lead to a 'deadlock'-like situation during master starting up:
> (1) The create namespace table procedure failed in the middle of master
> crash/failover. When master re-started, it tried to rollback, one step of
> rollback is to call QuotaManager to remove the table from NameSpaceQuota, but
> the QuotaManager has NOT started - so the rollback has to wait.
> (2). The QuotaManager would start in master after Namespace Manager starts.
> (3). The Namespace Manager is waiting for the table lock to be released by
> rollback of create namespace table procedure so that it can create namespace
> table as part of Namespace Manager initialization.
> {code}
> HMaster#finishActiveMasterInitialization() {
> ...
> status.setStatus("Starting namespace manager");
> initNamespace();
> ...
> status.setStatus("Starting quota manager");
> initQuotaManager();
> ...
> }
> {code}
> (4). Now (1) waits for (2), which waits for (3), which waits for (1) - no one
> make progress & master could not complete initialization and fails to come up.
> {noformat}
> 2015-05-28 10:01:26,890 INFO [ip-111-22-33-444:16000.activeMasterManager]
> master.TableNamespaceManager: Namespace table not found. Creating...
> 2015-05-28 10:06:22,016 WARN [ProcedureExecutorThread-0]
> procedure.CreateTableProcedure: Failed rollback attempt
> step=CREATE_TABLE_PRE_OPERATION table=hbase:namespace
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timed out while
> waiting on quota manager to be available
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitFor(ProcedureSyncWait.java:122)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitFor(ProcedureSyncWait.java:102)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.getMasterQuotaManager(ProcedureSyncWait.java:184)
> at
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteTableStates(DeleteTableProcedure.java:408)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.rollbackState(CreateTableProcedure.java:169)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.rollbackState(CreateTableProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:121)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:414)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:808)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:773)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:653)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:626)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:70)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:413)
> 2015-05-28 10:06:22,169 WARN [ProcedureExecutorThread-1]
> procedure.CreateTableProcedure: The table hbase:namespace does not exist in
> meta but has a znode. run hbck to fix inconsistencies.
> 2015-05-28 10:06:27,292 FATAL [ip-111-22-33-444:16000.activeMasterManager]
> master.HMaster: Failed to become active master
> java.io.IOException: Timedout 300000ms waiting for namespace table to be
> assigned
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
> at
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:980)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1632)
> at java.lang.Thread.run(Thread.java:745)
> 2015-05-28 10:06:27,293 FATAL [ip-111-22-33-444:16000.activeMasterManager]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor]
> 2015-05-28 10:06:27,293 FATAL [ip-111-22-33-444:16000.activeMasterManager]
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 300000ms waiting for namespace table to be
> assigned
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
> at
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:980)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1632)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
