[
https://issues.apache.org/jira/browse/HDFS-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638367#comment-17638367
]
ASF GitHub Bot commented on HDFS-16839:
---------------------------------------
tasanuma commented on code in PR #5128:
URL: https://github.com/apache/hadoop/pull/5128#discussion_r1031730865
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java:
##########
@@ -957,6 +957,58 @@ public void testSkipReconstructionWithManyBusyNodes2() {
assertNull(work);
}
+ @Test
+ public void testSkipReconstructionWithManyBusyNodes3() {
+ NameNode.initMetrics(new Configuration(),
HdfsServerConstants.NamenodeRole.NAMENODE);
+ long blockId = -9223372036854775776L; // real ec block id
+ // RS-3-2 EC policy
+ ErasureCodingPolicy ecPolicy =
+ SystemErasureCodingPolicies.getPolicies().get(1);
+
+ // create an EC block group: 3 data blocks + 2 parity blocks
+ Block aBlockGroup = new Block(blockId, ecPolicy.getCellSize() *
ecPolicy.getNumDataUnits(), 0);
+ BlockInfoStriped aBlockInfoStriped = new BlockInfoStriped(aBlockGroup,
ecPolicy);
+
+ // create 4 storageInfo, which means 1 block is missing
+ DatanodeStorageInfo ds1 = DFSTestUtil.createDatanodeStorageInfo(
+ "storage1", "1.1.1.1", "rack1", "host1");
+ DatanodeStorageInfo ds2 = DFSTestUtil.createDatanodeStorageInfo(
+ "storage2", "2.2.2.2", "rack2", "host2");
+ DatanodeStorageInfo ds3 = DFSTestUtil.createDatanodeStorageInfo(
+ "storage3", "3.3.3.3", "rack3", "host3");
+ DatanodeStorageInfo ds4 = DFSTestUtil.createDatanodeStorageInfo(
+ "storage4", "4.4.4.4", "rack4", "host4");
+
+ // link block with storage
+ aBlockInfoStriped.addStorage(ds1, aBlockGroup);
+ aBlockInfoStriped.addStorage(ds2, new Block(blockId + 1, 0, 0));
+ aBlockInfoStriped.addStorage(ds3, new Block(blockId + 2, 0, 0));
+ aBlockInfoStriped.addStorage(ds4, new Block(blockId + 3, 0, 0));
+
+ addEcBlockToBM(blockId, ecPolicy);
+ aBlockInfoStriped.setBlockCollectionId(mockINodeId);
+
+ // reconstruction should be scheduled
+ BlockReconstructionWork work =
bm.scheduleReconstruction(aBlockInfoStriped, 3);
+ assertNotNull(work);
+
+ ExtendedBlock dummyBlock = new ExtendedBlock("bpid", 1, 1, 1);
+ DatanodeDescriptor dummyDD = ds1.getDatanodeDescriptor();
+ DatanodeDescriptor[] dummyDDArray = new DatanodeDescriptor[]{dummyDD};
+ DatanodeStorageInfo[] dummyDSArray = new DatanodeStorageInfo[]{ds1};
+ // simulate the 2 nodes reach maxReplicationStreams
+ for(int i = 0; i < bm.maxReplicationStreams; i++){ //Add some dummy EC
reconstruction task
+ ds3.getDatanodeDescriptor().addBlockToBeErasureCoded(dummyBlock,
dummyDDArray,
+ dummyDSArray, new byte[0], new byte[0], ecPolicy);
Review Comment:
Other than the issues that @tomscut mentioned, it looks good to me too.
> It should consider EC reconstruction work when we determine if a node is busy
> -----------------------------------------------------------------------------
>
> Key: HDFS-16839
> URL: https://issues.apache.org/jira/browse/HDFS-16839
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kidd5368
> Assignee: Kidd5368
> Priority: Major
> Labels: pull-request-available
>
> In chooseSourceDatanodes( ), I think it's more reasonable if we take EC
> reconstruction work as a consideration when we determine if a node is busy or
> not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
