[
https://issues.apache.org/jira/browse/HDFS-15359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120484#comment-17120484
]
Vinayakumar B commented on HDFS-15359:
--------------------------------------
Thanks [~ayushtkn] for the patch.
I think the approach of allowing commited block only in case of write happened
to all nodes is very reasonable to prevent unexpected dataloss.
2 minor comments
{code}
if (b.isStriped()) {
BlockInfoStriped blkStriped = (BlockInfoStriped) b;
if (b.getUnderConstructionFeature().getExpectedStorageLocations().length
!= blkStriped.getRealTotalBlockNum()) {
return b + " is a striped block in " + state + " with less then "
+ "required number of blocks.";
}
}
{code}
Move this check after `if (state != BlockUCState.COMMITTED) ` check. It makes
more sense there.
In test,
{code}
// Check if the blockgroup isn't complete then file close shouldn't be
// success with block in committed state.
cluster.getDataNodes().get(0).shutdown();
FSDataOutputStream str = dfs.create(new Path("/dir/file1"));
for (int i = 0; i < 1024 * 1024 * 4; i++) {
str.write(i);
}
DataNodeTestUtils.pauseIBR(cluster.getDataNodes().get(0));
DataNodeTestUtils.pauseIBR(cluster.getDataNodes().get(1));
LambdaTestUtils.intercept(IOException.class, "", () -> str.close());
{code}
You should `pauseIBR` datanodes 1 and 2. 0 is already shutdown.
+1 once addessed.
> EC: Allow closing a file with committed blocks
> ----------------------------------------------
>
> Key: HDFS-15359
> URL: https://issues.apache.org/jira/browse/HDFS-15359
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
> Reporter: Ayush Saxena
> Assignee: Ayush Saxena
> Priority: Major
> Attachments: HDFS-15359-01.patch, HDFS-15359-02.patch,
> HDFS-15359-03.patch, HDFS-15359-04.patch
>
>
> Presently, {{dfs.namenode.file.close.num-committed-allowed}} is ignored in
> case of EC blocks. But in case of heavy loads, IBR's from Datanode may get
> delayed and cause the file write to fail. So, can allow EC files to close
> with blocks in committed state as REP files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
