[
https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870749#action_12870749
]
Scott Carey commented on HDFS-1172:
-----------------------------------
Perhaps when the write pipeline completes, it should pass back the block
information so that the initial commit to the NN can atomically add all the
blocks.
Example:
DN's in pipe are DN1, DN2, DN3.
A block is being written, the client writes to DN1, which writes to DN2, which
writes to DN3. When DN3 completes, it notifies DN2 and provides its block
replica information. When DN2 completes and has DN3's response, it passes its
information, along with DN3's, to DN1. When DN1 completes, and has DN2's
information along with DN3's, it reports to the NN the information about all 3
replicas, and lastly returns to the original client.
This will have a few benefits:
Fewer RPC's to the NN, and therefore less NN load.
Atomic visibility of all replicas to the NN and clients.
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
> Key: HDFS-1172
> URL: https://issues.apache.org/jira/browse/HDFS-1172
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't
> find an existing JIRA. It often happens that we see the NN schedule
> replication on the last block of files very quickly after they're completed,
> before the other DNs in the pipeline have a chance to report the new block.
> This results in a lot of extra replication work on the cluster, as we
> replicate the block and then end up with multiple excess replicas which are
> very quickly deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
