[ https://issues.apache.org/jira/browse/HDFS-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107589#comment-15107589 ]
Jing Zhao commented on HDFS-8999: --------------------------------- Thanks for updating the patch, Nicholas! The current patch looks good to me. Some minors: # Will the current default retry policy retry the append RPC for RetriableException? In that case we do not need to retry explicitly in {{callAppend}}? # We can always use {{getMinStorageNum}} instead of adding {{BlockManager#getMinReplication}} since we already check if the block/file is striped. # In FSEditLogLoader, when applying OP_ClOSE, all the blocks are forced to close. Thus looks like we can blindly complete the file without bringing in the committed block logic. {code} - fsNamesys.leaseManager.removeLeases(Lists.newArrayList(file.getId())); - file.toCompleteFile(file.getModificationTime()); + fsNamesys.removeLeaseAndCompleteFile(file); {code} # We now have two places in {{completeFileInternal}} to add committed blocks into the pending queue: one is in {{commitOrCompleteLastBlock}}, and the other is {{addCommittedBlocksToPending}}. Do we still need the first one? Looks like we can now remove the changes from HDFS-1172 and HDFS-9535. {code} // commit the last block and complete it if it has minimum replicas fsn.commitOrCompleteLastBlock(pendingFile, iip, last); if (!fsn.checkFileProgress(src, pendingFile, true)) { return false; } fsn.addCommittedBlocksToPending(pendingFile); {code} > Namenode need not wait for {{blockReceived}} for the last block before > completing a file. > ----------------------------------------------------------------------------------------- > > Key: HDFS-8999 > URL: https://issues.apache.org/jira/browse/HDFS-8999 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode > Reporter: Jitendra Nath Pandey > Assignee: Tsz Wo Nicholas Sze > Attachments: h8999_20151228.patch, h8999_20160106.patch, > h8999_20160106b.patch, h8999_20160106c.patch, h8999_20160111.patch, > h8999_20160113.patch, h8999_20160114.patch > > > This comes out of a discussion in HDFS-8763. Pasting [~jingzhao]'s comment > from the jira: > {quote} > ...whether we need to let NameNode wait for all the block_received msgs to > announce the replica is safe. Looking into the code, now we have > # NameNode knows the DataNodes involved when initially setting up the > writing pipeline > # If any DataNode fails during the writing, client bumps the GS and > finally reports all the DataNodes included in the new pipeline to NameNode > through the updatePipeline RPC. > # When the client received the ack for the last packet of the block (and > before the client tries to close the file on NameNode), the replica has been > finalized in all the DataNodes. > Then in this case, when NameNode receives the close request from the client, > the NameNode already knows the latest replicas for the block. Currently the > checkReplication call only counts in all the replicas that NN has already > received the block_received msg, but based on the above #2 and #3, it may be > safe to also count in all the replicas in the > BlockUnderConstructionFeature#replicas? > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)