[ 
https://issues.apache.org/jira/browse/HDFS-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107589#comment-15107589
 ] 

Jing Zhao commented on HDFS-8999:
---------------------------------

Thanks for updating the patch, Nicholas! The current patch looks good to me. 
Some minors:
# Will the current default retry policy retry the append RPC for 
RetriableException? In that case we do not need to retry explicitly in 
{{callAppend}}?
# We can always use {{getMinStorageNum}} instead of adding 
{{BlockManager#getMinReplication}} since we already check if the block/file is 
striped.
# In FSEditLogLoader, when applying OP_ClOSE, all the blocks are forced to 
close. Thus looks like we can blindly complete the file without bringing in the 
committed block logic.
{code}
-        fsNamesys.leaseManager.removeLeases(Lists.newArrayList(file.getId()));
-        file.toCompleteFile(file.getModificationTime());
+        fsNamesys.removeLeaseAndCompleteFile(file);
{code}
# We now have two places in {{completeFileInternal}} to add committed blocks 
into the pending queue: one is in {{commitOrCompleteLastBlock}}, and the other 
is {{addCommittedBlocksToPending}}. Do we still need the first one? Looks like 
we can now remove the changes from HDFS-1172 and HDFS-9535.
{code}
    // commit the last block and complete it if it has minimum replicas
    fsn.commitOrCompleteLastBlock(pendingFile, iip, last);

    if (!fsn.checkFileProgress(src, pendingFile, true)) {
      return false;
    }

    fsn.addCommittedBlocksToPending(pendingFile);
{code}

> Namenode need not wait for {{blockReceived}} for the last block before 
> completing a file.
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-8999
>                 URL: https://issues.apache.org/jira/browse/HDFS-8999
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>            Reporter: Jitendra Nath Pandey
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h8999_20151228.patch, h8999_20160106.patch, 
> h8999_20160106b.patch, h8999_20160106c.patch, h8999_20160111.patch, 
> h8999_20160113.patch, h8999_20160114.patch
>
>
> This comes out of a discussion in HDFS-8763. Pasting [~jingzhao]'s comment 
> from the jira:
> {quote}
> ...whether we need to let NameNode wait for all the block_received msgs to 
> announce the replica is safe. Looking into the code, now we have
>    # NameNode knows the DataNodes involved when initially setting up the 
> writing pipeline
>    # If any DataNode fails during the writing, client bumps the GS and 
> finally reports all the DataNodes included in the new pipeline to NameNode 
> through the updatePipeline RPC.
>    # When the client received the ack for the last packet of the block (and 
> before the client tries to close the file on NameNode), the replica has been 
> finalized in all the DataNodes.
> Then in this case, when NameNode receives the close request from the client, 
> the NameNode already knows the latest replicas for the block. Currently the 
> checkReplication call only counts in all the replicas that NN has already 
> received the block_received msg, but based on the above #2 and #3, it may be 
> safe to also count in all the replicas in the 
> BlockUnderConstructionFeature#replicas?
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to