[
https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871385#action_12871385
]
Scott Carey commented on HDFS-1172:
-----------------------------------
bq. # Scott's solution of making the primary DN send the blockReceived on
account of all DNs would work, but sounds complicated, expecially in the
failure cases (eg what if the primary DN fails just before sending the RPC? Do
we lose all the replicas? No good!)
Yeah, its complicated. To simplify failure scenarios, leave the rest to be
similar to the current state -- the next regularly scheduled ping from a DN
will provide the new block information, but the primary DN will still do its
best to send all the block data it can gather so that the initial registration
is as complete as possible. Perhaps the NN treats this extra information as
provisional, until it gets a ping from the other DN's to confirm.
Functionally, this won't differ much from Dhruba's proposition, and is more
complicated.
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
> Key: HDFS-1172
> URL: https://issues.apache.org/jira/browse/HDFS-1172
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't
> find an existing JIRA. It often happens that we see the NN schedule
> replication on the last block of files very quickly after they're completed,
> before the other DNs in the pipeline have a chance to report the new block.
> This results in a lot of extra replication work on the cluster, as we
> replicate the block and then end up with multiple excess replicas which are
> very quickly deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
