[
https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009004#comment-13009004
]
Konstantin Shvachko commented on HDFS-1172:
-------------------------------------------
I think this should be controlled by "dfs.namenode.replication.interval". It is
currently set to 3 secs. If DNs do not keep up with reporting blocks it should
be increased.
Putting blocks to pendingReplication feels like a trick, although it slows down
replication of the last block.
I think the right solution would be to add logic to processing of a failed
pipeline. When this happens the client asks for a new generation stamp. At this
point NN can make a note that this block will not have enough replicas. This
will distinguish between blocks that have not been reported yet, and those that
will never be reported. This is much more work.
In practice I think tuning up the "replication.interval" parameter should be
sufficient.
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
> Key: HDFS-1172
> URL: https://issues.apache.org/jira/browse/HDFS-1172
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
> Attachments: HDFS-1172.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't
> find an existing JIRA. It often happens that we see the NN schedule
> replication on the last block of files very quickly after they're completed,
> before the other DNs in the pipeline have a chance to report the new block.
> This results in a lot of extra replication work on the cluster, as we
> replicate the block and then end up with multiple excess replicas which are
> very quickly deleted.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
