[ https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15676336#comment-15676336 ]
Nandakumar commented on HDFS-10206: ----------------------------------- bq. From the below code, it seems each level will increase by 2. Yes, you're right. Distance from a node to it's parent is assumed to be 1, so adding the value of both the nodes will make it 2. Sorry for the confusion on that. {quote} one is the reader being a datanode in a remote rack in a large cluster; for that NetworkTopology already has the reader in its tree, it will be faster to compare parents reference. {quote} We can use {{NetworkTopology.contains(node)}} to check if the reader is a datanode and use {{NetworkTopology.getDistance(node1, node2)}} to get the distance (which also calculates the distance by summing up the nodes distances to their closest common ancestor), but both of these methods use {{ReadWriteLock.readLock()}} which might again impact the performance. Any comments on using {{NetworkTopology.contains(node)}} to check and use {{NetworkTopology.getDistance(node1, node2)}} to get the distance in case if the reader is an off rack datanode? I'm currently working on the benchmarking, will update it once it's done. > getBlockLocations might not sort datanodes properly by distance > --------------------------------------------------------------- > > Key: HDFS-10206 > URL: https://issues.apache.org/jira/browse/HDFS-10206 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Ming Ma > Assignee: Nandakumar > Attachments: HDFS-10206.000.patch > > > If the DFSClient machine is not a datanode, but it shares its rack with some > datanodes of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}} > might not put the local-rack datanodes at the beginning of the sorted list. > That is because the function didn't call {{networktopology.add(client);}} to > properly set the node's parent node; something required by > {{networktopology.sortByDistance}} to compute distance between two nodes in > the same topology tree. > Another issue with {{networktopology.sortByDistance}} is it only > distinguishes local rack from remote rack, but it doesn't support general > distance calculation to tell how remote the rack is. > {noformat} > NetworkTopology.java > protected int getWeight(Node reader, Node node) { > // 0 is local, 1 is same rack, 2 is off rack > // Start off by initializing to off rack > int weight = 2; > if (reader != null) { > if (reader.equals(node)) { > weight = 0; > } else if (isOnSameRack(reader, node)) { > weight = 1; > } > } > return weight; > } > {noformat} > HDFS-10203 has suggested moving the sorting from namenode to DFSClient to > address another issue. Regardless of where we do the sorting, we still need > fix the issues outline here. > Note that BlockPlacementPolicyDefault shares the same NetworkTopology object > used by DatanodeManager and requires Nodes stored in the topology to be > {{DatanodeDescriptor}} for block placement. So we need to make sure we don't > pollute the NetworkTopology if we plan to fix it on the server side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org