[
https://issues.apache.org/jira/browse/HBASE-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363212#comment-16363212
]
stack commented on HBASE-19992:
-------------------------------
This might be my fault going between hbase1 and hbase2 with different
codebases. Leaving open for now. I thought it was the migration of hbase1 table
state from zk setting table as enabled and so 'existing' but something else
happened such that hbase:meta had no hbase:namespace mention. Leaving open for
now in case I see this again in testing.
> Hole in namespace table assign
> ------------------------------
>
> Key: HBASE-19992
> URL: https://issues.apache.org/jira/browse/HBASE-19992
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Major
>
> If the assign fails before it comes up in a Master initialization, the table
> will have been created and may even be marked ENABLED successfully, but on
> restart, we don't assign the table.
> Manifest is:
> {code}
> 2018-02-13 11:45:24,504 ERROR [master/ve0524:16000] master.HMaster: Failed to
> become active master
> java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table
> to be assigned and enabled: ENABLED
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
> at
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
> ... 4 more
> 2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: Master
> server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: *****
> ABORTING master ve0524.halxg.cloudera.com,16000,1518550812400: Unhandled
> exception. Starting shutdown. *****
> java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Caused by:
> java.io.IOException: Timedout 300000ms waiting for namespace table to be
> assigned and enabled: ENABLED
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
> at
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
>
>
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
> ... 4 more
> {code}
> Last thing in log before Master crash was:
> 2018-02-13 11:34:17,084 INFO [master/ve0524:16000] hbase.MetaTableAccessor:
> Updated table hbase:namespace state to ENABLED in META
> There is no one doing an assign subsequent to initial create table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
