[
https://issues.apache.org/jira/browse/HDFS-16088?focusedWorklogId=618882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618882
]
ASF GitHub Bot logged work on HDFS-16088:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Jul/21 05:57
Start Date: 06/Jul/21 05:57
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on a change in pull request #3140:
URL: https://github.com/apache/hadoop/pull/3140#discussion_r664253973
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java
##########
@@ -236,4 +241,93 @@ private void testBalancerWithObserver(boolean
withObserverFailure)
}
}
}
+
+ /**
+ * Comparing the results of getLiveDatanodeStorageReport()
+ * from the active and standby NameNodes,
+ * the results should be the same.
+ */
+ @Test(timeout = 60000)
+ public void testGetLiveDatanodeStorageReport() throws Exception {
+ Configuration conf = new HdfsConfiguration();
+ TestBalancer.initConf(conf);
+ assertEquals(TEST_CAPACITIES.length, TEST_RACKS.length);
+ NNConf nn1Conf = new MiniDFSNNTopology.NNConf("nn1");
+ nn1Conf.setIpcPort(HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT);
+ Configuration copiedConf = new Configuration(conf);
+ // Try capture NameNodeConnector log.
+ LogCapturer log =LogCapturer.captureLogs(
+ LoggerFactory.getLogger(NameNodeConnector.class));
+ // We needs to assert datanode info from ANN and SNN, so the
+ // heartbeat should disabled for the duration of method execution
+ copiedConf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 60000);
+ cluster = new MiniDFSCluster.Builder(copiedConf)
+ .nnTopology(MiniDFSNNTopology.simpleHATopology())
+ .numDataNodes(TEST_CAPACITIES.length)
+ .racks(TEST_RACKS)
+ .simulatedCapacities(TEST_CAPACITIES)
+ .build();
+ HATestUtil.setFailoverConfigurations(cluster, conf);
+ try {
+ cluster.waitActive();
+ cluster.transitionToActive(0);
+ URI namenode = (URI) DFSUtil.getInternalNsRpcUris(conf)
+ .toArray()[0];
+ String nsId = DFSUtilClient.getNameServiceIds(conf)
+ .toArray()[0].toString();
+
+ // request to active namenode
+ NameNodeConnector nncActive = new NameNodeConnector(
+ "nncActive", namenode,
+ nsId, new Path("/test"),
+ null, conf, NameNodeConnector.DEFAULT_MAX_IDLE_ITERATIONS);
+ DatanodeStorageReport[] ldspFromAnn =
Review comment:
`ldspFromAnn` here is not very explicit IMO, is `datanodeStorageReports`
more clear here?
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java
##########
@@ -236,4 +241,93 @@ private void testBalancerWithObserver(boolean
withObserverFailure)
}
}
}
+
+ /**
+ * Comparing the results of getLiveDatanodeStorageReport()
+ * from the active and standby NameNodes,
+ * the results should be the same.
+ */
+ @Test(timeout = 60000)
+ public void testGetLiveDatanodeStorageReport() throws Exception {
+ Configuration conf = new HdfsConfiguration();
+ TestBalancer.initConf(conf);
+ assertEquals(TEST_CAPACITIES.length, TEST_RACKS.length);
+ NNConf nn1Conf = new MiniDFSNNTopology.NNConf("nn1");
+ nn1Conf.setIpcPort(HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT);
+ Configuration copiedConf = new Configuration(conf);
+ // Try capture NameNodeConnector log.
+ LogCapturer log =LogCapturer.captureLogs(
+ LoggerFactory.getLogger(NameNodeConnector.class));
+ // We needs to assert datanode info from ANN and SNN, so the
+ // heartbeat should disabled for the duration of method execution
+ copiedConf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 60000);
+ cluster = new MiniDFSCluster.Builder(copiedConf)
+ .nnTopology(MiniDFSNNTopology.simpleHATopology())
+ .numDataNodes(TEST_CAPACITIES.length)
+ .racks(TEST_RACKS)
+ .simulatedCapacities(TEST_CAPACITIES)
+ .build();
+ HATestUtil.setFailoverConfigurations(cluster, conf);
+ try {
+ cluster.waitActive();
+ cluster.transitionToActive(0);
+ URI namenode = (URI) DFSUtil.getInternalNsRpcUris(conf)
+ .toArray()[0];
+ String nsId = DFSUtilClient.getNameServiceIds(conf)
+ .toArray()[0].toString();
+
+ // request to active namenode
Review comment:
It is better to begin with uppercase character and end with period for
annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 618882)
Time Spent: 2h 50m (was: 2h 40m)
> Standby NameNode process getLiveDatanodeStorageReport request to reduce
> Active load
> -----------------------------------------------------------------------------------
>
> Key: HDFS-16088
> URL: https://issues.apache.org/jira/browse/HDFS-16088
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: tomscut
> Assignee: tomscut
> Priority: Major
> Labels: pull-request-available
> Attachments: standyby-ipcserver.jpg
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> As with HDFS-13183, NameNodeConnector#getLiveDatanodeStorageReport() can also
> request to SNN to reduce the ANN load.
> There are two points that need to be mentioned:
> 1. FSNamesystem#getDatanodeStorageReport() is OperationCategory.UNCHECKED,
> so we can access SNN directly.
> 2. We can share the same UT(testBalancerRequestSBNWithHA) with
> NameNodeConnector#getBlocks().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
