[
https://issues.apache.org/jira/browse/HADOOP-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565781#action_12565781
]
butlermh edited comment on HADOOP-2767 at 2/5/08 8:08 AM:
-------------------------------------------------------------
Hairong, thanks for the speedy reply.
As you note, looking at the code, DFS does not call this to exclude nodes. So
it is not a problem for DFS. Specifically excluding racks happens in
NetworkTopology.chooseRandom(String) while excluding nodes occurs in
ReplicationTargetChooser.chooseRandom(int numReplicas, String nodes, List<Node>
excludedNodes).
However you have written the patch, so my suggestion is we fix this. I took a
look at the patch and there is a small error in it - it said
if (excludedNode instanceof InnerNode) {
numOfExcludedLeaves = ((InnerNode)excludedNode).getNumOfLeaves();
isLeaf = true;
}
but if it is an inner node, it can't be a leaf? Consequently it fails the test
so I think it should be
if (excludedNode instanceof InnerNode) {
numOfExcludedLeaves = ((InnerNode)excludedNode).getNumOfLeaves();
} else { isLeaf = true; }
I've produced an updated patch and also modified my test code and incorporated
it into TestNetworkTopology.java. This should resolve the issue.
Thanks again!
was (Author: butlermh):
Hairong, thanks for the speedy reply.
As you note, looking at the code, DFS does not call this to exclude nodes. So
it is not a problem for DFS. Specifically excluding racks happens in
NetworkTopology.chooseRandom(String) while excluding nodes occurs in
ReplicationTargetChooser.chooseRandom(int numReplicas, String nodes, List<Node>
excludedNodes).
However you have written the patch, so my suggestion is we fix this. I took a
look at the patch and there is a small error in it - it said
if (excludedNode instanceof InnerNode) {
numOfExcludedLeaves = ((InnerNode)excludedNode).getNumOfLeaves();
isLeaf = true;
}
but if it is an inner node, it can't be a leaf? Consequently it fails the test
so I think it should be
if (excludedNode instanceof InnerNode) {
numOfExcludedLeaves = ((InnerNode)excludedNode).getNumOfLeaves();
} else {
isLeaf = true;
}
I've produced an updated patch and also modified my test code and incorporated
it into TestNetworkTopology.java. This should resolve the issue.
Thanks again!
Unfortunately the patch does not fix the problem for excluding nodes, but as
this is not used in DFS, my suggestion is to close this issue.
I am going to submit an updated
> org.apache.hadoop.net.NetworkTopology.InnerNode#getLeaf does not return the
> last node on a rack when used with an excluded node
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2767
> URL: https://issues.apache.org/jira/browse/HADOOP-2767
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Mark Butler
> Assignee: Hairong Kuang
> Priority: Minor
> Attachments: excludedLeaf.patch, excludedLeaf2.patch,
> NetworkTopologyTest.java
>
>
> I have written some test code that shows NetworkTopology.InnerNode#getLeaf
> will never return the last node on the rack if it is called with an
> excludedNode (for example the first node on the rack).
> Consequently I suspect that NetworkTopology.chooseRandom() will never returns
> the last node on the remote rack for the second replica in DFS.
> I have some test code that demonstrates this problem at the getLeaf level,
> although it is necessary to change the visibility of the
> NetworkTopology.InnerNode, NetworkTopology.InnerNode#getLeaf and
> NetworkTopology.getNode from private to package default to run the test.
> TODO: Demonstrate problem at NetworkTopology.chooseRandom level, then submit
> candidate fix.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
