[
http://issues.apache.org/jira/browse/HADOOP-692?page=comments#action_12448822 ]
Doug Cutting commented on HADOOP-692:
-------------------------------------
We need an extensible API for describing network topology & distance. Here's
stab at one.
public Interface NetworkLocatable {
/** Return the names of each network level of this node: host, rack,
datacenter, etc. */
String[] getNetworkLocation();
}
It is tempting to use something more abstract, e.g. a distance method, but that
would not provide a concise indication of the topology, and would require
linear scans, or storage of a full distance matrix, when, e.g., placing blocks.
As a first implementation, I like the idea of trying to distribute blocks to
increasing network levels: first to the same host, then to the same rack, then
to the same datacenter, etc. until the replication level is exhausted. If the
replication level is higher than the number of levels, then a uniform
distribution at the highest level should be sought.
Down the road we may wish to make this strategy extensible, or to have a more
elaborate distance calculation, but we can first probably get a lot of milage
from this simple model.
> Rack-aware Replica Placement
> ----------------------------
>
> Key: HADOOP-692
> URL: http://issues.apache.org/jira/browse/HADOOP-692
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.8.0
> Reporter: Hairong Kuang
> Assigned To: Hairong Kuang
> Fix For: 0.9.0
>
>
> This issue assumes that HDFS runs on a cluster of computers that spread
> across many racks. Communication between two nodes on different racks needs
> to go through switches. Bandwidth in/out of a rack may be less than the total
> bandwidth of machines in the rack. The purpose of rack-aware replica
> placement is to improve data reliability, availability, and network bandwidth
> utilization. The basic idea is that each data node determines to which rack
> it belongs at the startup time and notifies the name node of the rack id upon
> registration. The name node maintains a rackid-to-datanode map and tries to
> place replicas across racks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
