[
https://issues.apache.org/jira/browse/HBASE-18058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013479#comment-16013479
]
Allan Yang edited comment on HBASE-18058 at 5/17/17 4:13 AM:
-------------------------------------------------------------
{quote}
Normally in this case RegionServer will crash due to zookeeper session timeout,
similar like when RS full GC, right? Mind share the case in your scenario? How
do you keep RS alive while zookeeper down for some while? Thanks. Allan Yang
{quote}
Yes, It is a very interesting case and really happened. If the server hosting
zookeeper is disk full, the zookeeper quorum won't really went down but reject
all write request. So at HBase side, new zk write request will suffers from
exception and retry. But connection remains so the session won't timeout. When
disk full situation have been resolved, the zookeeper quorum can work normally
again. But the very high sleep time cause some module of RegionServer/HMaster
will still sleep for a long time(in our case, the balancer) before working.
was (Author: allan163):
{quote}
Normally in this case RegionServer will crash due to zookeeper session timeout,
similar like when RS full GC, right? Mind share the case in your scenario? How
do you keep RS alive while zookeeper down for some while? Thanks. Allan Yang
{quote}
Yes, It is a very interesting case and really happened. If the server hosting
zookeeper is disk full, the zookeeper quorum won't really went down but reject
all write request. So at HBase side, new zk write request will suffers from
exception and retry. But connection remains so the session won't timeout. When
disk full situation have been resolved, the zookeeper quorum can work normally
again. But the very high sleep time cause some module of RegionServer will
still sleep for a long time(in our case, the balancer) before working.
> Zookeeper retry sleep time should have a up limit
> -------------------------------------------------
>
> Key: HBASE-18058
> URL: https://issues.apache.org/jira/browse/HBASE-18058
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.0.0, 1.4.0
> Reporter: Allan Yang
> Assignee: Allan Yang
> Attachments: HBASE-18058-branch-1.patch,
> HBASE-18058-branch-1.v2.patch, HBASE-18058.patch
>
>
> Now, in {{RecoverableZooKeeper}}, the retry backoff sleep time grow
> exponentially, but it doesn't have any up limit. It directly lead to a long
> long recovery time after Zookeeper going down for some while and come back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
