[
https://issues.apache.org/jira/browse/HADOOP-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576454#action_12576454
]
dhruba borthakur commented on HADOOP-2656:
------------------------------------------
The Datanode needs to store a block generation stamp for each block. The
original idea was to store the block generation stamp inside the meta file of
each block. One major disadvantage of this approach is that generation of a
block report requires that each meta file be opened and the generation stamp
read from it. With about 50K blocks per datanode and a seek time of 10 ms, this
might require 300 seconds. This means that the time to restart a cluster cannot
be lesser than this 300 seconds.
A few other alternatives:
1. Encode the generation stamp into the name of the metafile. Each metafile
will look like blkxxxxxx.genstamp.meta. The block file will remain the same.
2. Encode the generation stamp into the name of the block file. Each block file
will be of the form blkxxxxxx.genstamp. The metafile will remain the same.
3. Encode the generation stamp into the name of a new zero-size file named
blkxxxxx.genstamp. The block file and the metadata file will remain the same.
4. A completely separate file (one per datanode) that records the metadata of
all blocks in the datanode.
I propose that we implement option 1.
> Support for upgrading existing cluster to facilitate appends to HDFS files
> --------------------------------------------------------------------------
>
> Key: HADOOP-2656
> URL: https://issues.apache.org/jira/browse/HADOOP-2656
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> HADOOP-1700 describes the design for supporting appends to HDFS files. This
> design requires a distributed-upgrade to existing cluster installations. The
> design specifies that the DataNode persist the 8-byte BlockGenerationStamp in
> the block metadata file. The upgrade code will introduce this new field in
> the block metadata file and initialize this value to 0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
