[
https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885844#action_12885844
]
Rodrigo Schmidt commented on HDFS-1094:
---------------------------------------
@Hairong: Just a few observations
1) The policy I'm implementing doesn't fix the number of racks to 3 and I don't
think anyone agrees that should be the case. Dhruba mentioned this static
3-rack policy in this discussion, but I guess that was just an example of how
block placement can reduce the probability of failure.
2) There is a clear trade-off here. The more you restrict where you can place
blocks, the worse your maximum bandwidth and load balance will tend to be.
(because you will have fewer ways to rearrange blocks to maximize these
metrics). So, no free lunch.
3) Still, in some cases, we might want to reduce bandwidth and load balance
possibilities for a lower probability of data loss, specially if HDFS is being
used to store important data, backed up by some middleware that has a cache
layer.
Having said that, the patch I'm working on is a configurable policy where
configuration variables define windows in two spaces, racks and machines within
a rack, in which blocks can be placed. Potentially, these windows can be set to
be the entire cluster, which is exactly the default block placement policy. In
the other extreme, such windows can be defined to be as restrict as only three
machines, which will give the lowest probability of data loss but no freedom to
improve load balance or bandwidth.
Please let me know if this is not clear enough.
> Intelligent block placement policy to decrease probability of block loss
> ------------------------------------------------------------------------
>
> Key: HDFS-1094
> URL: https://issues.apache.org/jira/browse/HDFS-1094
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: prob.pdf, prob.pdf
>
>
> The current HDFS implementation specifies that the first replica is local and
> the other two replicas are on any two random nodes on a random remote rack.
> This means that if any three datanodes die together, then there is a
> non-trivial probability of losing at least one block in the cluster. This
> JIRA is to discuss if there is a better algorithm that can lower probability
> of losing a block.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
