[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631053#comment-13631053 ] Hudson commented on YARN-412: - Integrated in Hadoop-Mapreduce-trunk #1398 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1398/]) YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.5-beta > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631027#comment-13631027 ] Hudson commented on YARN-412: - Integrated in Hadoop-Hdfs-trunk #1371 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1371/]) YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470) YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than its host:port during scheduling which caused incorrect locality for containers. Contributed by Roger Hoover. (Revision 1467244) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.5-beta > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631006#comment-13631006 ] Hudson commented on YARN-412: - Integrated in Hadoop-Yarn-trunk #182 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/182/]) YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470) YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than its host:port during scheduling which caused incorrect locality for containers. Contributed by Roger Hoover. (Revision 1467244) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.5-beta > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630649#comment-13630649 ] Hudson commented on YARN-412: - Integrated in Hadoop-trunk-Commit #3610 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3610/]) YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.5-beta > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630075#comment-13630075 ] Hudson commented on YARN-412: - Integrated in Hadoop-Mapreduce-trunk #1397 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1397/]) YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than its host:port during scheduling which caused incorrect locality for containers. Contributed by Roger Hoover. (Revision 1467244) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.4-alpha > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630008#comment-13630008 ] Hudson commented on YARN-412: - Integrated in Hadoop-trunk-Commit #3607 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3607/]) YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than its host:port during scheduling which caused incorrect locality for containers. Contributed by Roger Hoover. (Revision 1467244) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621759#comment-13621759 ] Roger Hoover commented on YARN-412: --- [~acmurthy], I've got the patch back in shape. Can you please review it or let me know what the next step is? > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621735#comment-13621735 ] Hadoop QA commented on YARN-412: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576914/YARN-412.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/669//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/669//console This message is automatically generated. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621709#comment-13621709 ] Hitesh Shah commented on YARN-412: -- @Roger, for future reference ( may not be applicable to this jira ), it is good to leave earlier patch attachments lying around and not delete them when uploading newer patches. This can be used to trace review comments/feedback etc. As for hadoop-common, mvn eclipse:eclipse, it can be ignored for now. It is a known issue with an open jira that has not been addressed yet. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621338#comment-13621338 ] Hadoop QA commented on YARN-412: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576838/YARN-412.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:red}-1 eclipse:eclipse{color}. The patch failed to build with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/665//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/665//console This message is automatically generated. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619064#comment-13619064 ] Hadoop QA commented on YARN-412: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576418/YARN-412.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/638//console This message is automatically generated. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619047#comment-13619047 ] Roger Hoover commented on YARN-412: --- [~acmurthy] OK. I've regenerated the patch based on the top of tree. Finger crossed, it still passes all QA checks. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618433#comment-13618433 ] Arun C Murthy commented on YARN-412: [~theduderog] - No, you deserve credit for finding and fixing this. Thanks! > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch, YARN-412.patch, YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585224#comment-13585224 ] Hadoop QA commented on YARN-412: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570627/YARN-412.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 tests included appear to have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/424//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/424//console This message is automatically generated. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch, YARN-412.patch, YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584473#comment-13584473 ] Hadoop QA commented on YARN-412: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570495/YARN-412.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 tests included appear to have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:red}-1 eclipse:eclipse{color}. The patch failed to build with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/417//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/417//console This message is automatically generated. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch, YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583522#comment-13583522 ] Hadoop QA commented on YARN-412: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570348/YARN-412.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:red}-1 one of tests included doesn't have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/416//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/416//console This message is automatically generated. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583480#comment-13583480 ] Hadoop QA commented on YARN-412: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570342/YARN-412.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:red}-1 one of tests included doesn't have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/415//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/415//console This message is automatically generated. > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Priority: Minor > Labels: patch > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira