[
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9311:
--------------------------------
Attachment: HDFS-9311.002.patch
[~jnp], thank you for the review. I agree. The handler ratio was a holdover
from splitting away from the HDFS-9239 patch. For the scope of this issue, we
don't need that many threads.
I'm uploading patch v002 with these changes:
# Adjusted the threads as per Jitendra's feedback.
# Corrected some mocking that was causing failures in
{{TestZKFailoverController}} and {{TestZKFailoverControllerStress}} in the last
Jenkins run.
# Corrected Checkstyle and whitespace warning.
> Support optional offload of NameNode HA service health checks to a separate
> RPC server.
> ---------------------------------------------------------------------------------------
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion
> of the RPC handler pools (both client-facing and service-facing).
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers
> a failover. Depending on fencing configuration, the former active NameNode
> may be killed. In an overloaded situation, the new active NameNode is likely
> to suffer the same fate, because client load patterns don't change after the
> failover. This can degenerate into flapping between the 2 NameNodes without
> real recovery. If a NameNode had been killed by fencing, then it would have
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for
> isolating the HA health checks. These health checks are lightweight
> operations that do not suffer from contention issues on the namesystem lock
> or other shared resources. Isolating the RPC handlers is sufficient to avoid
> this situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
