[ https://issues.apache.org/jira/browse/HDFS-15359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120484#comment-17120484 ]
Vinayakumar B commented on HDFS-15359: -------------------------------------- Thanks [~ayushtkn] for the patch. I think the approach of allowing commited block only in case of write happened to all nodes is very reasonable to prevent unexpected dataloss. 2 minor comments {code} if (b.isStriped()) { BlockInfoStriped blkStriped = (BlockInfoStriped) b; if (b.getUnderConstructionFeature().getExpectedStorageLocations().length != blkStriped.getRealTotalBlockNum()) { return b + " is a striped block in " + state + " with less then " + "required number of blocks."; } } {code} Move this check after `if (state != BlockUCState.COMMITTED) ` check. It makes more sense there. In test, {code} // Check if the blockgroup isn't complete then file close shouldn't be // success with block in committed state. cluster.getDataNodes().get(0).shutdown(); FSDataOutputStream str = dfs.create(new Path("/dir/file1")); for (int i = 0; i < 1024 * 1024 * 4; i++) { str.write(i); } DataNodeTestUtils.pauseIBR(cluster.getDataNodes().get(0)); DataNodeTestUtils.pauseIBR(cluster.getDataNodes().get(1)); LambdaTestUtils.intercept(IOException.class, "", () -> str.close()); {code} You should `pauseIBR` datanodes 1 and 2. 0 is already shutdown. +1 once addessed. > EC: Allow closing a file with committed blocks > ---------------------------------------------- > > Key: HDFS-15359 > URL: https://issues.apache.org/jira/browse/HDFS-15359 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding > Reporter: Ayush Saxena > Assignee: Ayush Saxena > Priority: Major > Attachments: HDFS-15359-01.patch, HDFS-15359-02.patch, > HDFS-15359-03.patch, HDFS-15359-04.patch > > > Presently, {{dfs.namenode.file.close.num-committed-allowed}} is ignored in > case of EC blocks. But in case of heavy loads, IBR's from Datanode may get > delayed and cause the file write to fail. So, can allow EC files to close > with blocks in committed state as REP files -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org