[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-13 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631053#comment-13631053
 ] 

Hudson commented on YARN-412:
-

Integrated in Hadoop-Mapreduce-trunk #1398 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1398/])
YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470)

 Result = SUCCESS
acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Fix For: 2.0.5-beta
>
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-13 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631027#comment-13631027
 ] 

Hudson commented on YARN-412:
-

Integrated in Hadoop-Hdfs-trunk #1371 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1371/])
YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470)
YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than 
its host:port during scheduling which caused incorrect locality for containers. 
Contributed by Roger Hoover. (Revision 1467244)

 Result = FAILURE
acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt

acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java


> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Fix For: 2.0.5-beta
>
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-13 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631006#comment-13631006
 ] 

Hudson commented on YARN-412:
-

Integrated in Hadoop-Yarn-trunk #182 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/182/])
YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470)
YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than 
its host:port during scheduling which caused incorrect locality for containers. 
Contributed by Roger Hoover. (Revision 1467244)

 Result = SUCCESS
acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt

acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java


> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Fix For: 2.0.5-beta
>
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630649#comment-13630649
 ] 

Hudson commented on YARN-412:
-

Integrated in Hadoop-trunk-Commit #3610 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3610/])
YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470)

 Result = SUCCESS
acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Fix For: 2.0.5-beta
>
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630075#comment-13630075
 ] 

Hudson commented on YARN-412:
-

Integrated in Hadoop-Mapreduce-trunk #1397 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1397/])
YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather 
than its host:port during scheduling which caused incorrect locality for 
containers. Contributed by Roger Hoover. (Revision 1467244)

 Result = SUCCESS
acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java


> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Fix For: 2.0.4-alpha
>
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-12 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630008#comment-13630008
 ] 

Hudson commented on YARN-412:
-

Integrated in Hadoop-trunk-Commit #3607 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3607/])
YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather 
than its host:port during scheduling which caused incorrect locality for 
containers. Contributed by Roger Hoover. (Revision 1467244)

 Result = SUCCESS
acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java


> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-03 Thread Roger Hoover (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621759#comment-13621759
 ] 

Roger Hoover commented on YARN-412:
---

[~acmurthy], I've got the patch back in shape.  Can you please review it or let 
me know what the next step is?


> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-03 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621735#comment-13621735
 ] 

Hadoop QA commented on YARN-412:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12576914/YARN-412.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/669//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/669//console

This message is automatically generated.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-03 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621709#comment-13621709
 ] 

Hitesh Shah commented on YARN-412:
--

@Roger, for future reference ( may not be applicable to this jira ), it is good 
to leave earlier patch attachments lying around and not delete them when 
uploading newer patches. This can be used to trace review comments/feedback etc.

As for hadoop-common, mvn eclipse:eclipse, it can be ignored for now. It is a 
known issue with an open jira that has not been addressed yet.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-03 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621338#comment-13621338
 ] 

Hadoop QA commented on YARN-412:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12576838/YARN-412.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 eclipse:eclipse{color}.  The patch failed to build with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/665//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/665//console

This message is automatically generated.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-01 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619064#comment-13619064
 ] 

Hadoop QA commented on YARN-412:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12576418/YARN-412.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/638//console

This message is automatically generated.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-04-01 Thread Roger Hoover (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619047#comment-13619047
 ] 

Roger Hoover commented on YARN-412:
---

[~acmurthy] OK.  I've regenerated the patch based on the top of tree.  Finger 
crossed, it still passes all QA checks.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-03-31 Thread Arun C Murthy (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618433#comment-13618433
 ] 

Arun C Murthy commented on YARN-412:


[~theduderog] - No, you deserve credit for finding and fixing this. Thanks!

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Assignee: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch, YARN-412.patch, YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-02-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585224#comment-13585224
 ] 

Hadoop QA commented on YARN-412:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12570627/YARN-412.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 tests included appear to have a timeout.{color}

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/424//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/424//console

This message is automatically generated.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch, YARN-412.patch, YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-02-22 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584473#comment-13584473
 ] 

Hadoop QA commented on YARN-412:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12570495/YARN-412.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 tests included appear to have a timeout.{color}

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:red}-1 eclipse:eclipse{color}.  The patch failed to build with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/417//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/417//console

This message is automatically generated.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch, YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-02-21 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583522#comment-13583522
 ] 

Hadoop QA commented on YARN-412:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12570348/YARN-412.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

  {color:red}-1 one of tests included doesn't have a timeout.{color}

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/416//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/416//console

This message is automatically generated.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality

2013-02-21 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583480#comment-13583480
 ] 

Hadoop QA commented on YARN-412:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12570342/YARN-412.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

  {color:red}-1 one of tests included doesn't have a timeout.{color}

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/415//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/415//console

This message is automatically generated.

> FifoScheduler incorrectly checking for node locality
> 
>
> Key: YARN-412
> URL: https://issues.apache.org/jira/browse/YARN-412
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Roger Hoover
>Priority: Minor
>  Labels: patch
> Attachments: YARN-412.patch
>
>
> In the FifoScheduler, the assignNodeLocalContainers method is checking if the 
> data is local to a node by searching for the nodeAddress of the node in the 
> set of outstanding requests for the app.  This seems to be incorrect as it 
> should be checking hostname instead.  The offending line of code is 455:
> application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
> Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses 
> are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
> In the CapacityScheduler, it's done using hostname.  See 
> LeafQueue.assignNodeLocalContainers, line 1129
> application.getResourceRequest(priority, node.getHostName());
> Note that this bug does not affect the actual scheduling decisions made by 
> the FifoScheduler because even though it incorrect determines that a request 
> is not local to the node, it will still schedule the request immediately 
> because it's rack-local.  However, this bug may be adversely affecting the 
> reporting of job status by underreporting the number of tasks that were node 
> local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira