[
https://issues.apache.org/jira/browse/HADOOP-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665972#action_12665972
]
Raghu Angadi commented on HADOOP-5034:
--------------------------------------
+1 for the solution and I think this should go into 0.19 and up.. if not 0.18.
Main reason is that when we are affected by this issue, there is no work
around. It is pretty much a dead-lock : e.g. : if the cluster is almost ful,
the replications don't succeed, but if admin wants make room by deleting file,
blocks won't be deleted since there are replication pending.
Regd implementation, the patch adds new datanode heartbeat command that
combines invalidates and replications. I think it will simpler if we make
heartbeat reply to contain list or array of commands. This way we don't need
new command types.
> NameNode should send both both replication and deletion requests to DataNode
> in one reply to a heartbeat
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5034
> URL: https://issues.apache.org/jira/browse/HADOOP-5034
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: blockTransferInvalidate.patch
>
>
> Currently NameNode favors block replication requests over deletion requests.
> On reply to a heartbeat, NameNode does not send a block deletion request
> unless there is no block replication request.
> This brings a problem when a near-full cluster loses a bunch of DataNodes. In
> react to the DataNode loss, NameNode starts to replicate blocks. However,
> replication takes a lot of cpu and a lot of replications fail because of the
> lack of disk space. So the administrator tries to delete some DFS files to
> free up space. However, block deletion requests get delayed for very long
> time because it takes a long time to drain the block replication requests for
> most DataNodes.
> I'd like to propose to let NameNode to send both replication requests and
> deletion requests to DataNodes in one reply to a heartbeat. This also implies
> that the replication monitor should schedule both replication and deletion
> work in one iteration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
