[
https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190706#comment-14190706
]
Colin Patrick McCabe commented on HDFS-3107:
--------------------------------------------
bq. And it needs to be atomic e.g. not involving 5 RPC calls, otherwise
recovery would be a nightmare.
The block-copying solution could be done with 1 RPC. The NN receives the
truncate RPC. It marks the file as open-for-write, and creates a new block ID
for the last block. Then it returns to the client. Later, when the DNs which
have the last block check in with the NN via heartbeats, the NN sends them a
(new) command that leads them to duplicate the block file with a new ID. Then
when the NN receives word that these blocks have been duplicated, it closes the
file and truncate is complete.
This is pretty similar to the original patch, except the original patch had the
NN send a command that led the DNs to truncate the block on-disk (destroying
the part of it after the truncation point). Just like in the original patch,
the client may need to call {{isFileClosed}} a few times to wait for the
truncation to finish.
> HDFS truncate
> -------------
>
> Key: HDFS-3107
> URL: https://issues.apache.org/jira/browse/HDFS-3107
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Reporter: Lei Chang
> Assignee: Plamen Jeliazkov
> Attachments: HDFS-3107.008.patch, HDFS-3107.patch, HDFS-3107.patch,
> HDFS-3107.patch, HDFS-3107.patch, HDFS-3107.patch, HDFS-3107.patch,
> HDFS-3107.patch, HDFS_truncate.pdf, HDFS_truncate.pdf,
> HDFS_truncate_semantics_Mar15.pdf, HDFS_truncate_semantics_Mar21.pdf,
> editsStored, editsStored, editsStored.xml
>
> Original Estimate: 1,344h
> Remaining Estimate: 1,344h
>
> Systems with transaction support often need to undo changes made to the
> underlying storage when a transaction is aborted. Currently HDFS does not
> support truncate (a standard Posix operation) which is a reverse operation of
> append, which makes upper layer applications use ugly workarounds (such as
> keeping track of the discarded byte range per file in a separate metadata
> store, and periodically running a vacuum process to rewrite compacted files)
> to overcome this limitation of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
