[
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068657#comment-16068657
]
stack commented on HBASE-14190:
-------------------------------
Good point [~yangzhe1991].
[~allan163] yeah... would need to put aside the namespace table assign if meta
is not online... probably inside in the meta open handler. Would need to not
block either as per Phils comment. Not sure that possible in current setup in
branch-1.
> Assign system tables ahead of user region assignment
> ----------------------------------------------------
>
> Key: HBASE-14190
> URL: https://issues.apache.org/jira/browse/HBASE-14190
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Reporter: Ted Yu
> Assignee: Ted Yu
> Priority: Critical
> Attachments: 14190-system-wal-v1.txt, 14190-v12.4.txt, 14190-v12.txt
>
>
> Currently the namespace table region is assigned like user regions.
> I spent several hours working with a customer where master couldn't finish
> initialization.
> Even though master was restarted quite a few times, it went down with the
> following:
> {code}
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager]
> master.HMaster: Master server abort: loaded coprocessors are: []
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager]
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 300000ms waiting for namespace table to be
> assigned
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
> at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> During previous run(s), namespace table was created, hence leaving an entry
> in hbase:meta.
> The following if block in TableNamespaceManager#start() was skipped:
> {code}
> if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
> TableName.NAMESPACE_TABLE_NAME)) {
> {code}
> TableNamespaceManager#start() spins, waiting for namespace region to be
> assigned.
> There was issue in master assigning user regions.
> We tried issuing 'assign' command from hbase shell which didn't work because
> of the following check in MasterRpcServices#assignRegion():
> {code}
> master.checkInitialized();
> {code}
> This scenario can be avoided if we assign hbase:namespace table after
> hbase:meta is assigned but before user table region assignment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
