Ming Ma created HDFS-10206: ------------------------------ Summary: getBlockLocations might not sort datanodes properly by distance Key: HDFS-10206 URL: https://issues.apache.org/jira/browse/HDFS-10206 Project: Hadoop HDFS Issue Type: Bug Reporter: Ming Ma
If the DFSClient machine is not a datanode, but it shares its rack with some datanodes of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}} might not put the local-rack datanodes at the beginning of the sorted list. That is because the function didn't call {{networktopology.add(client);}} to properly set the node's parent node; something required by {{networktopology.sortByDistance}} to compute distance between two nodes in the same topology tree. Another issue with {{networktopology.sortByDistance}} is it only distinguishes local rack from remote rack, but it doesn't support general distance calculation to tell how remote the rack is. {noformat} NetworkTopology.java protected int getWeight(Node reader, Node node) { // 0 is local, 1 is same rack, 2 is off rack // Start off by initializing to off rack int weight = 2; if (reader != null) { if (reader.equals(node)) { weight = 0; } else if (isOnSameRack(reader, node)) { weight = 1; } } return weight; } {noformat} HDFS-10203 has suggested moving the sorting from namenode to DFSClient to address another issue. Regardless of where we do the sorting, we still fix the issues outline here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)