[ https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135818#comment-14135818 ]
Colin Patrick McCabe commented on HDFS-3107: -------------------------------------------- bq. However, an extremely common use-case is failed appends, where the writing application dies / is killed / etc, which results in writing corrupted or incomplete data to HDFS. The big problem here is that most HDFS guides suggest storing large amounts of data in each file (e.g. in a typical log-file setup, one simply appends each day's new data onto a much larger existing file). By not having a truncate command, this is essentially not supported. Most HDFS users have logic to handle partial records, due to this problem. For example, Flume can roll to a new file when the old one has an error. It's pretty simple to prefix your records with a length, and simply ignore partial records that result from an incomplete flush to a file. bq. Append should never have been implemented without truncate. I don't see what any of this has to do with append, since this issue could equally well come up without HDFS append. Remember that HDFS append really means "reopen for write." HDFS can have an error on writing and write a partial record even without anyone reopening for write. bq. This is not a particularly good use case, certainly not enough to justify the change we are discussing here. You can't protect users from every little potential accident. Indeed. As [~sureshms] already pointed out, adding truncate opens up the possibility of accidental truncation, which is worse than accidental appending. Also, in general, we have not seen users complain about accidentally appending data to a file. I can't even think of one support request or email about this issue. On the other hand, we have seen many cases where people have accidentally deleted things, and ask us to try to get the data back. In general, the main use-case I see here is better fuse and NFS support. > HDFS truncate > ------------- > > Key: HDFS-3107 > URL: https://issues.apache.org/jira/browse/HDFS-3107 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode > Reporter: Lei Chang > Assignee: Plamen Jeliazkov > Attachments: HDFS-3107.patch, HDFS_truncate_semantics_Mar15.pdf, > HDFS_truncate_semantics_Mar21.pdf > > Original Estimate: 1,344h > Remaining Estimate: 1,344h > > Systems with transaction support often need to undo changes made to the > underlying storage when a transaction is aborted. Currently HDFS does not > support truncate (a standard Posix operation) which is a reverse operation of > append, which makes upper layer applications use ugly workarounds (such as > keeping track of the discarded byte range per file in a separate metadata > store, and periodically running a vacuum process to rewrite compacted files) > to overcome this limitation of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)