[
https://issues.apache.org/jira/browse/HDFS-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934316#comment-13934316
]
Konstantin Shvachko commented on HDFS-6087:
-------------------------------------------
Not sure I fully understood what you propose. So please feel free to correct if
I am wrong.
# Sounds like you propose to update blockID every time the pipeline fails and
that will guarantee block immutability.
Isn't that similar to how current HDFS uses generationStamp? When pipeline
fails HDFS increments genStamp making previously created replicas outdated.
# Seems you propose to introduce an extra commitBlock() call to NN.
Current HDFS has similar logic. Block commit is incorporated with addBlock()
and complete() calls. E.g. addBlock() changes state to committed of the
previous block of the file and then allocates the new one.
# Don't see how you get rid of lease recovery. The purpose of which is to
reconcile different replicas of the incomplete last block, as they can have
different lengths or genStamps on different DNs, as the results of the client
or DNs failure in the middle of a data transfer.
If you propose to discard uncommitted blocks entirely, then it will break
current semantics, which states that if a byte was read by a client once it
should be readable by other clients as well.
# I guess it boils down to that your diagrams show regular work-flow, but don't
consider failure scenarios.
> Unify HDFS write/append/truncate
> --------------------------------
>
> Key: HDFS-6087
> URL: https://issues.apache.org/jira/browse/HDFS-6087
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Guo Ruijing
> Attachments: HDFS Design Proposal.pdf
>
>
> In existing implementation, HDFS file can be appended and HDFS block can be
> reopened for append. This design will introduce complexity including lease
> recovery. If we design HDFS block as immutable, it will be very simple for
> append & truncate. The idea is that HDFS block is immutable if the block is
> committed to namenode. If the block is not committed to namenode, it is HDFS
> client’s responsibility to re-added with new block ID.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
