[
https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949123#comment-14949123
]
Jing Zhao commented on HDFS-1172:
---------------------------------
Thanks for updating the patch, [~iwasakims]. The 012 patch looks good to me.
Some minors:
# Since every time when NN receives a block_received msg it will check and
update the pendingReplication queue for the corresponding block, it may be fine
to apply the same updating-pending-queue logic to all the blocks of a file.
Thus can we also pass true to {{storeAllocatedBlock}}?
# Instead of checking if the file is striped here, we can check if the block is
striped inside of {{BlockManager#commitOrCompleteLastBlock}}. And in this way
maybe we do not need the {{completeFile}} argument (the the above comment also
stands).
{code}
if (!blockManager.commitOrCompleteLastBlock(
fileINode, commitBlock, !fileINode.isStriped() && completeFile)) {
{code}
# In {{addExpectedReplicasToPending}}, maybe we can simplify the code by first
adding pending replicas into a list (instead of an array) and converting the
list into an array in the end. In this way, this part of code does not depend
on the logic that "all the current reported storages should be included in the
expected storage list".
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
> Key: HDFS-1172
> URL: https://issues.apache.org/jira/browse/HDFS-1172
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
> Assignee: Masatake Iwasaki
> Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch,
> HDFS-1172.009.patch, HDFS-1172.010.patch, HDFS-1172.011.patch,
> HDFS-1172.012.patch, HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt,
> replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't
> find an existing JIRA. It often happens that we see the NN schedule
> replication on the last block of files very quickly after they're completed,
> before the other DNs in the pipeline have a chance to report the new block.
> This results in a lot of extra replication work on the cluster, as we
> replicate the block and then end up with multiple excess replicas which are
> very quickly deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
