[ https://issues.apache.org/jira/browse/HDFS-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948814#comment-13948814 ]
Konstantin Shvachko commented on HDFS-6087: ------------------------------------------- > Currently, block recovery is very complicated during pipeline broken Indeed the complexity starts when something brakes, and I still don't see how you propose to simplify the process. But truncate as discussed in HDFS-3107 is performed on a closed file and therefore is much less related to pipeline recovery since there is no pipeline after the file is closed. Which for me makes truncate a simpler task than the rewrite of the pipeline. Nicholas in [his comment|https://issues.apache.org/jira/browse/HDFS-3107?focusedCommentId=13235941&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13235941] mentioned three ideas to implement truncate. I was recently thinking about another one. What if we implement truncate similar to lease recovery. That is when a client asks to truncate a file, NN changes the list of blocks by deleting some of the tail ones and decrementing the size of the last. Then NN issues a DataNodeCommand to recover the last block. DNs as the result of the recovery will truncated their replica files, and then call commitBockSynchronization() to report the new length to the NN. Sorry, didn't want to hijack your jira, so if you intend to proceed with more general design here I'll re-post my idea under HDFS-3107. > Unify HDFS write/append/truncate > -------------------------------- > > Key: HDFS-6087 > URL: https://issues.apache.org/jira/browse/HDFS-6087 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client > Reporter: Guo Ruijing > Attachments: HDFS Design Proposal.pdf, HDFS Design Proposal_3_14.pdf > > > In existing implementation, HDFS file can be appended and HDFS block can be > reopened for append. This design will introduce complexity including lease > recovery. If we design HDFS block as immutable, it will be very simple for > append & truncate. The idea is that HDFS block is immutable if the block is > committed to namenode. If the block is not committed to namenode, it is HDFS > client’s responsibility to re-added with new block ID. -- This message was sent by Atlassian JIRA (v6.2#6252)