[ https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14978245#comment-14978245 ]
Hudson commented on HDFS-9311: ------------------------------ FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #593 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/593/]) HDFS-9311. Support optional offload of NameNode HA service health checks (cnauroth: rev bf8e45298218f70e38838152f69c7705d8606bd6) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitorWithDedicatedHealthAddress.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java > Support optional offload of NameNode HA service health checks to a separate > RPC server. > --------------------------------------------------------------------------------------- > > Key: HDFS-9311 > URL: https://issues.apache.org/jira/browse/HDFS-9311 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode > Reporter: Chris Nauroth > Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, > HDFS-9311.003.patch > > > When a NameNode is overwhelmed with load, it can lead to resource exhaustion > of the RPC handler pools (both client-facing and service-facing). > Eventually, this blocks the health check RPC issued from ZKFC, which triggers > a failover. Depending on fencing configuration, the former active NameNode > may be killed. In an overloaded situation, the new active NameNode is likely > to suffer the same fate, because client load patterns don't change after the > failover. This can degenerate into flapping between the 2 NameNodes without > real recovery. If a NameNode had been killed by fencing, then it would have > to transition through safe mode, further delaying time to recovery. > This issue proposes a separate, optional RPC server at the NameNode for > isolating the HA health checks. These health checks are lightweight > operations that do not suffer from contention issues on the namesystem lock > or other shared resources. Isolating the RPC handlers is sufficient to avoid > this situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)