[
https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333295#comment-16333295
]
stack commented on HBASE-19794:
-------------------------------
While Jira was down I spent some time on this last night. The backup Master
tries to become active during cluster shutdown but only gets this far:
{code:java}
78612 Thread 1542 (M:1;asf903:32967):
78613 State: TIMED_WAITING
78614 Blocked count: 178
78615 Waited count: 389
78616 Stack:
78617 java.lang.Object.wait(Native Method)
78618
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:168)
78619 org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
78620 org.apache.hadoop.hbase.client.HTable.get(HTable.java:362)
78621
org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1117)
78622
org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:1960)
78623
org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.getTableState(ConnectionUtils.java:131)
78624
org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:573)
78625
org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.isTableDisabled(ConnectionUtils.java:131)
78626
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:219)
78627
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105)
78628 org.apache.hadoop.hbase.client.HTable.get(HTable.java:388)
78629 org.apache.hadoop.hbase.client.HTable.get(HTable.java:362)
78630
org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:139)
78631
org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:276)
78632
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:101)
78633
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:62)
78634
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
78635
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1059)
78636
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:921){code}
The backup Master will just be stuck here until all retries have been
exhausted. This is a variant on a issue seen elsewhere where client hosted in
server is trying to contact a server or region that is not going to show up,
usually because cluster is going down. We need means of signaling the client it
should give up because its host is going away. We probably also need to move
client communication off the main thread so the main thread remains available
and can react to shutdown.
Concurrent w/ my digging [~Apache9] was digging too and arrived at same place
(offline because Jira was down). He came up w/ a better workaround for now than
my cutting down on retries. He suggested minihbasecluster should put down
backup master's first, before we do the active Master (Thinking on it, it may
not work... damage may already have been done before we get to the shutdown
sequence... The backup master may have already started in on the shutdown
sequence).
Let me work up a patch based on Duo's
[https://github.com/Apache9/hbase/commit/97e030584504cc6019ef06462f6d44ca40125c45]
Let me add timeout, Duo's suggestion, and some other cleanup I came across
digging last night. Will also file issue to deal better w/ the root problem of
clients stuck in retry though cluster has been asked go down.
> TestZooKeeper hangs
> -------------------
>
> Key: HBASE-19794
> URL: https://issues.apache.org/jira/browse/HBASE-19794
> Project: HBase
> Issue Type: Bug
> Reporter: Duo Zhang
> Assignee: stack
> Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt
>
>
> Seems like the TestZKAsyncRegistry that hangs in shutdown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
