[
https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134206#comment-14134206
]
Colin Patrick McCabe commented on HDFS-3107:
--------------------------------------------
[~shv], [~zero45], do you have a design doc for this? It's a pretty big change
to HDFS semantics (previously, file data was immutable once written.)
At minimum, we need to decide:
* What the consistency semantics are (if some client reads a file at position X
after it has been truncted to X-1, what might the client see?) If data is
appended after a truncate, it seems like we could get divergent histories in
many situations, where one client sees one timeline and another client sees
another. Is this acceptable? If so, we'll need to describe what the behavior
actually is. Are we going to guarantee that files never appear to shrink while
clients have them open? How does this interact with hflush and hsync?
* How this interacts with snapshots. When truncating a block and then
re-appending, a divergent block will be created, invalidating previous
snapshots. It seems that someone will have to copy this block, so the question
is when. I see some discussion of that here in the comments, but I don't see
any conclusions.
* What are the use-cases. Right now we have discussed only one use-case:
allowing users to remove data they accidentially appended. I would add one
more use-case: improving support for fuse-dfs and NFS, which both require
truncate to be implemented. Are there any additional use-cases?
* And, of course... how it's going to be implemented :)
I think this should be a branch since as the HDFS append experience showed, it
may take some time to get everything right.
> HDFS truncate
> -------------
>
> Key: HDFS-3107
> URL: https://issues.apache.org/jira/browse/HDFS-3107
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Reporter: Lei Chang
> Assignee: Plamen Jeliazkov
> Attachments: HDFS-3107.patch, HDFS_truncate_semantics_Mar15.pdf,
> HDFS_truncate_semantics_Mar21.pdf
>
> Original Estimate: 1,344h
> Remaining Estimate: 1,344h
>
> Systems with transaction support often need to undo changes made to the
> underlying storage when a transaction is aborted. Currently HDFS does not
> support truncate (a standard Posix operation) which is a reverse operation of
> append, which makes upper layer applications use ugly workarounds (such as
> keeping track of the discarded byte range per file in a separate metadata
> store, and periodically running a vacuum process to rewrite compacted files)
> to overcome this limitation of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
