[ https://issues.apache.org/jira/browse/HBASE-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363212#comment-16363212 ]
stack commented on HBASE-19992: ------------------------------- This might be my fault going between hbase1 and hbase2 with different codebases. Leaving open for now. I thought it was the migration of hbase1 table state from zk setting table as enabled and so 'existing' but something else happened such that hbase:meta had no hbase:namespace mention. Leaving open for now in case I see this again in testing. > Hole in namespace table assign > ------------------------------ > > Key: HBASE-19992 > URL: https://issues.apache.org/jira/browse/HBASE-19992 > Project: HBase > Issue Type: Bug > Reporter: stack > Assignee: stack > Priority: Major > > If the assign fails before it comes up in a Master initialization, the table > will have been created and may even be marked ENABLED successfully, but on > restart, we don't assign the table. > Manifest is: > {code} > 2018-02-13 11:45:24,504 ERROR [master/ve0524:16000] master.HMaster: Failed to > become active master > java.lang.IllegalStateException: Expected the service > ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED > at > org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345) > at > org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291) > at > org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052) > at > org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916) > at > org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table > to be assigned and enabled: ENABLED > at > org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107) > at > org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62) > at > org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226) > at > org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050) > ... 4 more > 2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: Master > server abort: loaded coprocessors are: > [org.apache.hadoop.hbase.security.access.AccessController] > 2018-02-13 11:45:24,506 ERROR [master/ve0524:16000] master.HMaster: ***** > ABORTING master ve0524.halxg.cloudera.com,16000,1518550812400: Unhandled > exception. Starting shutdown. ***** > java.lang.IllegalStateException: Expected the service > ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED > at > org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345) > at > org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291) > at > org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1052) > at > org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:916) > at > org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555) > at java.lang.Thread.run(Thread.java:748) > > > Caused by: > java.io.IOException: Timedout 300000ms waiting for namespace table to be > assigned and enabled: ENABLED > at > org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:107) > at > org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62) > at > org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226) > > > at > org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1050) > ... 4 more > {code} > Last thing in log before Master crash was: > 2018-02-13 11:34:17,084 INFO [master/ve0524:16000] hbase.MetaTableAccessor: > Updated table hbase:namespace state to ENABLED in META > There is no one doing an assign subsequent to initial create table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)