[
https://issues.apache.org/jira/browse/HBASE-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19992:
--------------------------
Description:
If the assign fails before it comes up in a Master initialization, the table
will have been created and may even be marked ENABLED successfully, but on
restart, we don't assign the table.
Manifest is:
{code}
2018-02-13 11:45:24,504 ERROR [master/ve0524:16000] master.HMaster: Failed to
become active master
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table
to be assigned and enabled: ENABLED
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
... 4 more
2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: Master
server abort: loaded coprocessors are:
[org.apache.hadoop.hbase.security.access.AccessController]
2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: *****
ABORTING master ve0524.halxg.cloudera.com,16000,1518550812400: Unhandled
exception. Starting shutdown. *****
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException:
Timedout 300000ms waiting for namespace table to be assigned and enabled:
ENABLED
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
... 4 more
{code}
Last thing in log before Master crash was:
2018-02-13 11:34:17,084 INFO [master/ve0524:16000] hbase.MetaTableAccessor:
Updated table hbase:namespace state to ENABLED in META
There is no one doing an assign subsequent to initial create table.
was:
If the assign fails before it comes up in a Master initialization, the table
will have been created and may even be marked ENABLED successfully, but on
restart, we don't assign the table.
Manifest is:
{code}
2018-02-13 11:45:24,504 ERROR [master/ve0524:16000] master.HMaster: Failed to
become active master
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table
to be assigned and enabled: ENABLED
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
... 4 more
2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: Master
server abort: loaded coprocessors are:
[org.apache.hadoop.hbase.security.access.AccessController]
2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: *****
ABORTING master ve0524.halxg.cloudera.com,16000,1518550812400: Unhandled
exception. Starting shutdown. *****
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException:
Timedout 300000ms waiting for namespace table to be assigned and enabled:
ENABLED
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
... 4 more
{code}
Last thing in log before Master crash was:
2018-02-13 11:34:17,084 INFO [master/ve0524:16000] hbase.MetaTableAccessor:
Updated table hbase:namespace state to ENABLED in META
There is no one doing an assign.
> Hole in namespace table assign
> ------------------------------
>
> Key: HBASE-19992
> URL: https://issues.apache.org/jira/browse/HBASE-19992
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Major
>
> If the assign fails before it comes up in a Master initialization, the table
> will have been created and may even be marked ENABLED successfully, but on
> restart, we don't assign the table.
> Manifest is:
> {code}
> 2018-02-13 11:45:24,504 ERROR [master/ve0524:16000] master.HMaster: Failed to
> become active master
> java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table
> to be assigned and enabled: ENABLED
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
> at
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
> ... 4 more
> 2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: Master
> server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: *****
> ABORTING master ve0524.halxg.cloudera.com,16000,1518550812400: Unhandled
> exception. Starting shutdown. *****
> java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Caused by:
> java.io.IOException: Timedout 300000ms waiting for namespace table to be
> assigned and enabled: ENABLED
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107)
> at
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
>
>
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050)
> ... 4 more
> {code}
> Last thing in log before Master crash was:
> 2018-02-13 11:34:17,084 INFO [master/ve0524:16000] hbase.MetaTableAccessor:
> Updated table hbase:namespace state to ENABLED in META
> There is no one doing an assign subsequent to initial create table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
