[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)

2013-04-23 Thread Carlo Curino (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638825#comment-13638825
 ] 

Carlo Curino commented on YARN-569:
---

Regenerated patches based on conversations on YARN-45. We introduced the 
priority-first, containerid-second order of containers selection as per 
feedback we received, however I have some doubts. 

Looking at the 
[documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#Writing_an_ApplicationMaster]
 and at the MR use of priority in RMContainerAllocater.java (around line 98). I 
find the choice of priorities in MR seems not very amenable to preemption. In 
particular, MR assign the following priorities (where bigger value means less 
priority): 
# PRIORITY_MAP = 20 
# PRIORITY_REDUCE = 10
# PRIORITY_FAILED_MAP = 5  

I believe this is needed for slow_start parameter to make the desired effect, 
and to make sure the FAILED_MAPs to start before REDUCERs. 
However, if we use the (reverse) priority to instruct the choice of containers 
to preempt, we will preempt REDUCERs after MAPs which is I think wrong. 
I think this might unveil some issues with the semantics of Priority, as they 
do not capture a long-running value of this container for the AM, but 
rather value at time of allocation. To be sorted-out before commit. 


 CapacityScheduler: support for preemption (using a capacity monitor)
 

 Key: YARN-569
 URL: https://issues.apache.org/jira/browse/YARN-569
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: capacityscheduler
Reporter: Carlo Curino
Assignee: Carlo Curino
 Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf, 
 YARN-569.patch


 There is a tension between the fast-pace reactive role of the 
 CapacityScheduler, which needs to respond quickly to 
 applications resource requests, and node updates, and the more introspective, 
 time-based considerations 
 needed to observe and correct for capacity balance. To this purpose we opted 
 instead of hacking the delicate
 mechanisms of the CapacityScheduler directly to add support for preemption by 
 means of a Capacity Monitor,
 which can be run optionally as a separate service (much like the 
 NMLivelinessMonitor).
 The capacity monitor (similarly to equivalent functionalities in the fairness 
 scheduler) operates running on intervals 
 (e.g., every 3 seconds), observe the state of the assignment of resources to 
 queues from the capacity scheduler, 
 performs off-line computation to determine if preemption is needed, and how 
 best to edit the current schedule to 
 improve capacity, and generates events that produce four possible actions:
 # Container de-reservations
 # Resource-based preemptions
 # Container-based preemptions
 # Container killing
 The actions listed above are progressively more costly, and it is up to the 
 policy to use them as desired to achieve the rebalancing goals. 
 Note that due to the lag in the effect of these actions the policy should 
 operate at the macroscopic level (e.g., preempt tens of containers
 from a queue) and not trying to tightly and consistently micromanage 
 container allocations. 
 - Preemption policy  (ProportionalCapacityPreemptionPolicy): 
 - 
 Preemption policies are by design pluggable, in the following we present an 
 initial policy (ProportionalCapacityPreemptionPolicy) we have been 
 experimenting with.  The ProportionalCapacityPreemptionPolicy behaves as 
 follows:
 # it gathers from the scheduler the state of the queues, in particular, their 
 current capacity, guaranteed capacity and pending requests (*)
 # if there are pending requests from queues that are under capacity it 
 computes a new ideal balanced state (**)
 # it computes the set of preemptions needed to repair the current schedule 
 and achieve capacity balance (accounting for natural completion rates, and 
 respecting bounds on the amount of preemption we allow for each round)
 # it selects which applications to preempt from each over-capacity queue (the 
 last one in the FIFO order)
 # it remove reservations from the most recently assigned app until the amount 
 of resource to reclaim is obtained, or until no more reservations exits
 # (if not enough) it issues preemptions for containers from the same 
 applications (reverse chronological order, last assigned container first) 
 again until necessary or until no containers except the AM container are left,
 # (if not enough) it moves onto unreserve and preempt from the next 
 application. 
 # containers that have been asked to preempt are tracked across executions. 
 If a containers is among the one to be preempted for more than a certain 
 time, the container is moved in a 

[jira] [Updated] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently

2013-04-23 Thread nemon lou (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nemon lou updated YARN-276:
---

Attachment: YARN-276.patch

 Capacity Scheduler can hang when submit many jobs concurrently
 --

 Key: YARN-276
 URL: https://issues.apache.org/jira/browse/YARN-276
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Affects Versions: 3.0.0, 2.0.1-alpha
Reporter: nemon lou
Assignee: nemon lou
 Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch

   Original Estimate: 24h
  Remaining Estimate: 24h

 In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity 
 scheduler can hang with most resources taken up by AM and don't have enough 
 resources for tasks.And then all applications hang there.
 The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not 
 check directly.Instead ,this property only used for maxActiveApplications. 
 And maxActiveApplications is computed by minimumAllocation (not by Am 
 actually used).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638923#comment-13638923
 ] 

Hadoop QA commented on YARN-276:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/1258/YARN-276.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/804//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/804//console

This message is automatically generated.

 Capacity Scheduler can hang when submit many jobs concurrently
 --

 Key: YARN-276
 URL: https://issues.apache.org/jira/browse/YARN-276
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Affects Versions: 3.0.0, 2.0.1-alpha
Reporter: nemon lou
Assignee: nemon lou
 Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch

   Original Estimate: 24h
  Remaining Estimate: 24h

 In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity 
 scheduler can hang with most resources taken up by AM and don't have enough 
 resources for tasks.And then all applications hang there.
 The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not 
 check directly.Instead ,this property only used for maxActiveApplications. 
 And maxActiveApplications is computed by minimumAllocation (not by Am 
 actually used).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently

2013-04-23 Thread nemon lou (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638944#comment-13638944
 ] 

nemon lou commented on YARN-276:


Parameter maxActiveApplicationsPerUsers is changed to 
maxAMResourcePerQueuePerUserPercent.Also checked by user's actual AM used 
resources.
And the webService has this change,too.
[~tgraves]

 Capacity Scheduler can hang when submit many jobs concurrently
 --

 Key: YARN-276
 URL: https://issues.apache.org/jira/browse/YARN-276
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Affects Versions: 3.0.0, 2.0.1-alpha
Reporter: nemon lou
Assignee: nemon lou
 Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch

   Original Estimate: 24h
  Remaining Estimate: 24h

 In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity 
 scheduler can hang with most resources taken up by AM and don't have enough 
 resources for tasks.And then all applications hang there.
 The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not 
 check directly.Instead ,this property only used for maxActiveApplications. 
 And maxActiveApplications is computed by minimumAllocation (not by Am 
 actually used).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-549) YarnClient.submitApplication should wait for application to be accepted by the RM

2013-04-23 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638947#comment-13638947
 ] 

Hudson commented on YARN-549:
-

Integrated in Hadoop-Yarn-trunk #192 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/192/])
YARN-549. YarnClient.submitApplication should wait for application to be 
accepted by the RM (Zhijie Shen via bikas) (Revision 1470797)

 Result = SUCCESS
bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1470797
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YarnClient.submitApplication should wait for application to be accepted by 
 the RM
 -

 Key: YARN-549
 URL: https://issues.apache.org/jira/browse/YARN-549
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: Proposal of Asynchronous Application Submission_v1.pdf, 
 YARN-549.1.patch, YARN-549.2.patch, YARN-549.3.patch, YARN-549.4.patch


 Currently, when submitting an application, storeApplication will be called 
 for recovery. However, it is a blocking API, and is likely to block 
 concurrent application submissions. Therefore, it is good to make application 
 submission asynchronous, and postpone storeApplication. YarnClient needs to 
 change to wait for the whole operation to complete so that clients can be 
 notified after the application is really submitted. YarnClient needs to wait 
 for application to reach SUBMITTED state or beyond.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-583) Application cache files should be localized under local-dir/usercache/userid/appcache/appid/filecache

2013-04-23 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638949#comment-13638949
 ] 

Hudson commented on YARN-583:
-

Integrated in Hadoop-Yarn-trunk #192 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/192/])
YARN-583. Moved application level local resources to be localized under the 
filecache sub-directory under application directory. Contributed by Omkar Vinit 
Joshi. (Revision 1470812)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1470812
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java


 Application cache files should be localized under 
 local-dir/usercache/userid/appcache/appid/filecache
 -

 Key: YARN-583
 URL: https://issues.apache.org/jira/browse/YARN-583
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Omkar Vinit Joshi
Assignee: Omkar Vinit Joshi
 Fix For: 2.0.5-beta

 Attachments: yarn-583-20130416.1.patch, yarn-583-20130416.patch, 
 yarn-583-20130417.patch, yarn-583-20130419.patch, yarn-583-20130422.patch


 Currently application cache files are getting localized under 
 local-dir/usercache/userid/appcache/appid/. however they should be localized 
 under filecache sub directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently

2013-04-23 Thread nemon lou (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nemon lou updated YARN-276:
---

Labels: incompatible  (was: )

 Capacity Scheduler can hang when submit many jobs concurrently
 --

 Key: YARN-276
 URL: https://issues.apache.org/jira/browse/YARN-276
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Affects Versions: 3.0.0, 2.0.1-alpha
Reporter: nemon lou
Assignee: nemon lou
  Labels: incompatible
 Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, 
 YARN-276.patch, YARN-276.patch

   Original Estimate: 24h
  Remaining Estimate: 24h

 In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity 
 scheduler can hang with most resources taken up by AM and don't have enough 
 resources for tasks.And then all applications hang there.
 The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not 
 check directly.Instead ,this property only used for maxActiveApplications. 
 And maxActiveApplications is computed by minimumAllocation (not by Am 
 actually used).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-549) YarnClient.submitApplication should wait for application to be accepted by the RM

2013-04-23 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639005#comment-13639005
 ] 

Hudson commented on YARN-549:
-

Integrated in Hadoop-Hdfs-trunk #1381 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1381/])
YARN-549. YarnClient.submitApplication should wait for application to be 
accepted by the RM (Zhijie Shen via bikas) (Revision 1470797)

 Result = FAILURE
bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1470797
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YarnClient.submitApplication should wait for application to be accepted by 
 the RM
 -

 Key: YARN-549
 URL: https://issues.apache.org/jira/browse/YARN-549
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: Proposal of Asynchronous Application Submission_v1.pdf, 
 YARN-549.1.patch, YARN-549.2.patch, YARN-549.3.patch, YARN-549.4.patch


 Currently, when submitting an application, storeApplication will be called 
 for recovery. However, it is a blocking API, and is likely to block 
 concurrent application submissions. Therefore, it is good to make application 
 submission asynchronous, and postpone storeApplication. YarnClient needs to 
 change to wait for the whole operation to complete so that clients can be 
 notified after the application is really submitted. YarnClient needs to wait 
 for application to reach SUBMITTED state or beyond.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-583) Application cache files should be localized under local-dir/usercache/userid/appcache/appid/filecache

2013-04-23 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639007#comment-13639007
 ] 

Hudson commented on YARN-583:
-

Integrated in Hadoop-Hdfs-trunk #1381 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1381/])
YARN-583. Moved application level local resources to be localized under the 
filecache sub-directory under application directory. Contributed by Omkar Vinit 
Joshi. (Revision 1470812)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1470812
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java


 Application cache files should be localized under 
 local-dir/usercache/userid/appcache/appid/filecache
 -

 Key: YARN-583
 URL: https://issues.apache.org/jira/browse/YARN-583
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Omkar Vinit Joshi
Assignee: Omkar Vinit Joshi
 Fix For: 2.0.5-beta

 Attachments: yarn-583-20130416.1.patch, yarn-583-20130416.patch, 
 yarn-583-20130417.patch, yarn-583-20130419.patch, yarn-583-20130422.patch


 Currently application cache files are getting localized under 
 local-dir/usercache/userid/appcache/appid/. however they should be localized 
 under filecache sub directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT

2013-04-23 Thread Thomas Graves (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639056#comment-13639056
 ] 

Thomas Graves commented on YARN-126:


Thanks for working on this.  

A few comments. Unfortunately we aren't consistent across hadoop how we print 
usages.  

- In the usage print out, I'd prefer to separate out the generic options like 
-D, -conf, and -rm from the actual commands so that its more obvious to users 
what the commands are. 
- Capitalize usage in yarn rmadmin (Usage:)
- the usage should say something like: yarn rmadmin [generic options] COMMAND


We should also fix the GenericOptionsParser to not reference the jobtracker 
anymore, but I can file a separate jira for that.

 yarn rmadmin help message contains reference to hadoop cli and JT
 -

 Key: YARN-126
 URL: https://issues.apache.org/jira/browse/YARN-126
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Affects Versions: 2.0.3-alpha
Reporter: Thomas Graves
Assignee: Rémy SAISSY
  Labels: usability
 Attachments: YARN-126.patch


 has option to specify a job tracker and the last line for general command 
 line syntax had bin/hadoop command [genericOptions] [commandOptions]
 ran yarn rmadmin to get usage:
 RMAdmin
 Usage: java RMAdmin
[-refreshQueues]
[-refreshNodes]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshAdminAcls]
[-refreshServiceAcl]
[-help [cmd]]
 Generic options supported are
 -conf configuration file specify an application configuration file
 -D property=valueuse value for given property
 -fs local|namenode:port  specify a namenode
 -jt local|jobtracker:portspecify a job tracker
 -files comma separated list of filesspecify comma separated files to be 
 copied to the map reduce cluster
 -libjars comma separated list of jarsspecify comma separated jar files 
 to include in the classpath.
 -archives comma separated list of archivesspecify comma separated 
 archives to be unarchived on the compute machines.
 The general command line syntax is
 bin/hadoop command [genericOptions] [commandOptions]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-549) YarnClient.submitApplication should wait for application to be accepted by the RM

2013-04-23 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639084#comment-13639084
 ] 

Hudson commented on YARN-549:
-

Integrated in Hadoop-Mapreduce-trunk #1408 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1408/])
YARN-549. YarnClient.submitApplication should wait for application to be 
accepted by the RM (Zhijie Shen via bikas) (Revision 1470797)

 Result = SUCCESS
bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1470797
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 YarnClient.submitApplication should wait for application to be accepted by 
 the RM
 -

 Key: YARN-549
 URL: https://issues.apache.org/jira/browse/YARN-549
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.0.5-beta

 Attachments: Proposal of Asynchronous Application Submission_v1.pdf, 
 YARN-549.1.patch, YARN-549.2.patch, YARN-549.3.patch, YARN-549.4.patch


 Currently, when submitting an application, storeApplication will be called 
 for recovery. However, it is a blocking API, and is likely to block 
 concurrent application submissions. Therefore, it is good to make application 
 submission asynchronous, and postpone storeApplication. YarnClient needs to 
 change to wait for the whole operation to complete so that clients can be 
 notified after the application is really submitted. YarnClient needs to wait 
 for application to reach SUBMITTED state or beyond.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-583) Application cache files should be localized under local-dir/usercache/userid/appcache/appid/filecache

2013-04-23 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639086#comment-13639086
 ] 

Hudson commented on YARN-583:
-

Integrated in Hadoop-Mapreduce-trunk #1408 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1408/])
YARN-583. Moved application level local resources to be localized under the 
filecache sub-directory under application directory. Contributed by Omkar Vinit 
Joshi. (Revision 1470812)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1470812
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java


 Application cache files should be localized under 
 local-dir/usercache/userid/appcache/appid/filecache
 -

 Key: YARN-583
 URL: https://issues.apache.org/jira/browse/YARN-583
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Omkar Vinit Joshi
Assignee: Omkar Vinit Joshi
 Fix For: 2.0.5-beta

 Attachments: yarn-583-20130416.1.patch, yarn-583-20130416.patch, 
 yarn-583-20130417.patch, yarn-583-20130419.patch, yarn-583-20130422.patch


 Currently application cache files are getting localized under 
 local-dir/usercache/userid/appcache/appid/. however they should be localized 
 under filecache sub directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated

2013-04-23 Thread Timothy St. Clair (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639090#comment-13639090
 ] 

Timothy St. Clair commented on YARN-600:


What happens when a node does not have it enabled?  SetAffinity?

 Hook up cgroups CPU settings to the number of virtual cores allocated
 -

 Key: YARN-600
 URL: https://issues.apache.org/jira/browse/YARN-600
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: resourcemanager, scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza

 YARN-3 introduced CPU isolation and monitoring through cgroups.  YARN-2 and 
 introduced CPU scheduling in the capacity scheduler, and YARN-326 will 
 introduce it in the fair scheduler.  The number of virtual cores allocated to 
 a container should be used to weight the number of cgroups CPU shares given 
 to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-561:
---

Attachment: YARN-561.10.patch

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-289) Fair scheduler allows reservations that won't fit on node

2013-04-23 Thread Sandy Ryza (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated YARN-289:


Attachment: YARN-289-1.patch

 Fair scheduler allows reservations that won't fit on node
 -

 Key: YARN-289
 URL: https://issues.apache.org/jira/browse/YARN-289
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-289-1.patch, YARN-289.patch


 An application requests a container with 1024 MB.  It then requests a 
 container with 2048 MB.  A node shows up with 1024 MB available.  Even if the 
 application is the only one running, neither request will be scheduled on it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-289) Fair scheduler allows reservations that won't fit on node

2013-04-23 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639282#comment-13639282
 ] 

Sandy Ryza commented on YARN-289:
-

Uploaded patch to fix javadoc warnings

 Fair scheduler allows reservations that won't fit on node
 -

 Key: YARN-289
 URL: https://issues.apache.org/jira/browse/YARN-289
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-289-1.patch, YARN-289.patch


 An application requests a container with 1024 MB.  It then requests a 
 container with 2048 MB.  A node shows up with 1024 MB available.  Even if the 
 application is the only one running, neither request will be scheduled on it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639309#comment-13639309
 ] 

Hadoop QA commented on YARN-561:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580081/YARN-561.10.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 14 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/805//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/805//console

This message is automatically generated.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-289) Fair scheduler allows reservations that won't fit on node

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639320#comment-13639320
 ] 

Hadoop QA commented on YARN-289:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580088/YARN-289-1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/806//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/806//console

This message is automatically generated.

 Fair scheduler allows reservations that won't fit on node
 -

 Key: YARN-289
 URL: https://issues.apache.org/jira/browse/YARN-289
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-289-1.patch, YARN-289.patch


 An application requests a container with 1024 MB.  It then requests a 
 container with 2048 MB.  A node shows up with 1024 MB available.  Even if the 
 application is the only one running, neither request will be scheduled on it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639348#comment-13639348
 ] 

Hitesh Shah commented on YARN-561:
--

Comments:

ApplicationAttemptId appAttemptIdEnv - please use a more appropriate variable 
name.
System.getenv(Environment.CONTAINER_ID.name()) - what happens if the container 
id is not set in the env? 

{code}
 putEnvIfNotNull(environment, Environment.USER.name(), container.getUser());
{code}

Is the user allowed to override the user env var? 

{code}
+  LOG.info(Adding  + container.getContainer().getId()
   +  to application  + app.toString());
{code}
  - Doesn't the above happen so often that it will be a performance overhead? 
Should this be changed to DEBUG? @Vinod, any comments on this? 





 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639476#comment-13639476
 ] 

Vinod Kumar Vavilapalli commented on YARN-561:
--

bq. ApplicationAttemptId appAttemptIdEnv - please use a more appropriate 
variable name.
+1

bq. System.getenv(Environment.CONTAINER_ID.name()) - what happens if the 
container id is not set in the env?
Unless you are looking to run new MR APP against old YARN, this shouldn't 
happen because NM always sets it now, no?

bq. Is the user allowed to override the user env var? 
No. Mostly PWD too. But let's do those separately, this one dragged on for too 
long.

bq. Doesn't the above happen so often that it will be a performance overhead?
Should be fine for now, as there are other places like this. We should do a 
comprehensive log cleanup separately.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-392) Make it possible to schedule to specific nodes without dropping locality

2013-04-23 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639491#comment-13639491
 ] 

Alejandro Abdelnur commented on YARN-392:
-

The idea of a locality delay gives more flexibility, I like it. 

Though I think it may complicate things quite a bit on the scheduler to be able 
to do the right book-keeping. Today, because the delay is at app level there is 
not delay counting at allocation requests level.

If we move the delay at allocation request level, it means we'd have to keep a 
counter at 'rack' level that gets decremented on every allocation attempt and 
when hits zero we go rack if node was not fulfilled. 

This means that the allocation request in the scheduler will have to have a new 
delay-counter property.

The complexity comes that when an AM places a new allocation request, the AM 
must provide the non-empty allocation requests in full.

For example:

Requesting 5 containers for node1/rack1:

{code}
  location=* - containers=5
  location=rack1 - containers=5
  location=node1 - containers=5
{code}

Requesting 5 additional containers for node2/rack1 (with original allocation 
still pending):

{code}
  location=* - containers=10
  location=rack1 - containers=10
  location=node2 - containers=5
{code}

The current contract allows the scheduler just to put the */rack container 
requests without having to do a lookup and aggregation.

If we are now keeping a delay counter associated at */rack level and we do a 
put, we'll reset the delay-counter for the node1 request family. If we keep the 
delay-counter of node1 request family and use it for the node2 request family 
we'll be shorting the node2 request expected locality delay.

The delay-locality per container request has value but I think it may require 
much more work.

Given that, don't you think getting the current approach suggested/implemented 
by Arun/Sandy makes sense in the short term?

 Make it possible to schedule to specific nodes without dropping locality
 

 Key: YARN-392
 URL: https://issues.apache.org/jira/browse/YARN-392
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Sandy Ryza
 Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, 
 YARN-392.patch


 Currently its not possible to specify scheduling requests for specific nodes 
 and nowhere else. The RM automatically relaxes locality to rack and * and 
 assigns non-specified machines to the app.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT

2013-04-23 Thread JIRA

[ 
https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639513#comment-13639513
 ] 

Rémy SAISSY commented on YARN-126:
--

I removed the GenericOptionsParser because I saw that the yarn distributed 
shell ApplicationMaster.java relies on org.apache.commons.cli.GnuParser instead.
So I though that since an options parser is not something specific to Hadoop, 
it might be a good thing to remove the GenericOptionParser dependency where it 
is possible to favor the commons.cli library instead.
Isn't it the case?



 yarn rmadmin help message contains reference to hadoop cli and JT
 -

 Key: YARN-126
 URL: https://issues.apache.org/jira/browse/YARN-126
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Affects Versions: 2.0.3-alpha
Reporter: Thomas Graves
Assignee: Rémy SAISSY
  Labels: usability
 Attachments: YARN-126.patch


 has option to specify a job tracker and the last line for general command 
 line syntax had bin/hadoop command [genericOptions] [commandOptions]
 ran yarn rmadmin to get usage:
 RMAdmin
 Usage: java RMAdmin
[-refreshQueues]
[-refreshNodes]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshAdminAcls]
[-refreshServiceAcl]
[-help [cmd]]
 Generic options supported are
 -conf configuration file specify an application configuration file
 -D property=valueuse value for given property
 -fs local|namenode:port  specify a namenode
 -jt local|jobtracker:portspecify a job tracker
 -files comma separated list of filesspecify comma separated files to be 
 copied to the map reduce cluster
 -libjars comma separated list of jarsspecify comma separated jar files 
 to include in the classpath.
 -archives comma separated list of archivesspecify comma separated 
 archives to be unarchived on the compute machines.
 The general command line syntax is
 bin/hadoop command [genericOptions] [commandOptions]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639514#comment-13639514
 ] 

Vinod Kumar Vavilapalli commented on YARN-561:
--

The latest patch also removed testing of Environment.MALLOC_ARENA_MAX, but I 
believe the current test isn't really the right way neither is it sufficient. 
We should really be validating YarnConfiguration.NM_ADMIN_USER_ENV. Can you 
please file a separate ticket for that?

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639568#comment-13639568
 ] 

Hitesh Shah commented on YARN-561:
--

@Vinod, there are also checks in the MRAppMaster which pretty much validate 
everything that the NM is supposed to be setting in the env. ( look for callers 
of MRAppMaster#validateInputParam ). This can be probably addressed separately 
in any case.  

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-561:
---

Attachment: YARN-561.11.patch

This patch and last patch remove the testing of Environment.MALLOC_ARENA_MAX 
from TestContainerLaunch. Will file a new jira ticket for it.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, 
 YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, 
 YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-595) Refactor fair scheduler to use common Resources

2013-04-23 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639587#comment-13639587
 ] 

Alejandro Abdelnur commented on YARN-595:
-

patch looks good, couple of NITs:

* FairScheduler: patch is adding a bunch of System.out.println(), I guess this 
is debris from your testing, if we need to output any of this, please use the 
log.
* FSSchedulerNode: no need to initialize availableResource var on definition, 
it is always assigned in the constructor.

 Refactor fair scheduler to use common Resources
 ---

 Key: YARN-595
 URL: https://issues.apache.org/jira/browse/YARN-595
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-595.patch, YARN-595.patch


 resourcemanager.fair and resourcemanager.resources have two copies of 
 basically the same code for operations on Resource objects

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT

2013-04-23 Thread Thomas Graves (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639590#comment-13639590
 ] 

Thomas Graves commented on YARN-126:


{quote}Shouldn't this be the correct fix instead of throwing out 
GenericOptionsParser altogether?{quote}

I'm a bit torn on that. yes it does have a couple of the options used but it 
also has others that do not apply to the rmadmin command. It also has the 
generic usage statement of: bin/hadoop command [genericOptions] 
[commandOptions] which doesn't apply at all.  I guess another option is to 
change the GenericOptionsParser to be able to specify which options can be used 
and have it print the correct usage statements and so forth.  

Ideally I think we should redo all of the hadoop/hdfs/yarn/mapred commands to 
have consistent usage and interfaces but that is a much bigger change. 

 yarn rmadmin help message contains reference to hadoop cli and JT
 -

 Key: YARN-126
 URL: https://issues.apache.org/jira/browse/YARN-126
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Affects Versions: 2.0.3-alpha
Reporter: Thomas Graves
Assignee: Rémy SAISSY
  Labels: usability
 Attachments: YARN-126.patch


 has option to specify a job tracker and the last line for general command 
 line syntax had bin/hadoop command [genericOptions] [commandOptions]
 ran yarn rmadmin to get usage:
 RMAdmin
 Usage: java RMAdmin
[-refreshQueues]
[-refreshNodes]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshAdminAcls]
[-refreshServiceAcl]
[-help [cmd]]
 Generic options supported are
 -conf configuration file specify an application configuration file
 -D property=valueuse value for given property
 -fs local|namenode:port  specify a namenode
 -jt local|jobtracker:portspecify a job tracker
 -files comma separated list of filesspecify comma separated files to be 
 copied to the map reduce cluster
 -libjars comma separated list of jarsspecify comma separated jar files 
 to include in the classpath.
 -archives comma separated list of archivesspecify comma separated 
 archives to be unarchived on the compute machines.
 The general command line syntax is
 bin/hadoop command [genericOptions] [commandOptions]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-45) Scheduler feedback to AM to release containers

2013-04-23 Thread Karthik Kambatla (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639595#comment-13639595
 ] 

Karthik Kambatla commented on YARN-45:
--

Barely skimmed through the patch, it looks good. Noticed a few javadoc typos we 
might like to fix. Will try to get in a more detailed review soon.

 Scheduler feedback to AM to release containers
 --

 Key: YARN-45
 URL: https://issues.apache.org/jira/browse/YARN-45
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Chris Douglas
Assignee: Carlo Curino
 Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch


 The ResourceManager strikes a balance between cluster utilization and strict 
 enforcement of resource invariants in the cluster. Individual allocations of 
 containers must be reclaimed- or reserved- to restore the global invariants 
 when cluster load shifts. In some cases, the ApplicationMaster can respond to 
 fluctuations in resource availability without losing the work already 
 completed by that task (MAPREDUCE-4584). Supplying it with this information 
 would be helpful for overall cluster utilization [1]. To this end, we want to 
 establish a protocol for the RM to ask the AM to release containers.
 [1] http://research.yahoo.com/files/yl-2012-003.pdf

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-422) Add AM-NM client library

2013-04-23 Thread Zhijie Shen (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-422:
-

Attachment: proposal_v1.pdf

Some initial thoughts about creating AMNMClient. Your comments, please.

 Add AM-NM client library
 

 Key: YARN-422
 URL: https://issues.apache.org/jira/browse/YARN-422
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
 Attachments: proposal_v1.pdf


 Create a simple wrapper over the AM-NM container protocol to provide hide the 
 details of the protocol implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT

2013-04-23 Thread JIRA

[ 
https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639613#comment-13639613
 ] 

Rémy SAISSY commented on YARN-126:
--

| Ideally I think we should redo all of the hadoop/hdfs/yarn/mapred commands to 
have consistent usage and interfaces but that is a much bigger change.

Yes and the command line options of these tools might be confusing sometimes.
In my daily life I am an IT consultant and when I install/train clients on 
Hadoop, they very often encounter two issues when it comes to using the command 
line:
 - which options are optional for a specific command (-fs in yarn rmadmin?)
 - where does some arguments must be specified (-Dmapreduce.map.java.opts when 
submiting a job on command line using hadoop jar).

Maybe the redo could be done in an umbrella JIRA with several issues, one per 
project?


 yarn rmadmin help message contains reference to hadoop cli and JT
 -

 Key: YARN-126
 URL: https://issues.apache.org/jira/browse/YARN-126
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Affects Versions: 2.0.3-alpha
Reporter: Thomas Graves
Assignee: Rémy SAISSY
  Labels: usability
 Attachments: YARN-126.patch, YARN-126.patch


 has option to specify a job tracker and the last line for general command 
 line syntax had bin/hadoop command [genericOptions] [commandOptions]
 ran yarn rmadmin to get usage:
 RMAdmin
 Usage: java RMAdmin
[-refreshQueues]
[-refreshNodes]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshAdminAcls]
[-refreshServiceAcl]
[-help [cmd]]
 Generic options supported are
 -conf configuration file specify an application configuration file
 -D property=valueuse value for given property
 -fs local|namenode:port  specify a namenode
 -jt local|jobtracker:portspecify a job tracker
 -files comma separated list of filesspecify comma separated files to be 
 copied to the map reduce cluster
 -libjars comma separated list of jarsspecify comma separated jar files 
 to include in the classpath.
 -archives comma separated list of archivesspecify comma separated 
 archives to be unarchived on the compute machines.
 The general command line syntax is
 bin/hadoop command [genericOptions] [commandOptions]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639624#comment-13639624
 ] 

Hadoop QA commented on YARN-561:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580122/YARN-561.11.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 14 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/807//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/807//console

This message is automatically generated.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, 
 YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, 
 YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639627#comment-13639627
 ] 

Hadoop QA commented on YARN-126:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580129/YARN-126.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/808//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/808//console

This message is automatically generated.

 yarn rmadmin help message contains reference to hadoop cli and JT
 -

 Key: YARN-126
 URL: https://issues.apache.org/jira/browse/YARN-126
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Affects Versions: 2.0.3-alpha
Reporter: Thomas Graves
Assignee: Rémy SAISSY
  Labels: usability
 Attachments: YARN-126.patch, YARN-126.patch


 has option to specify a job tracker and the last line for general command 
 line syntax had bin/hadoop command [genericOptions] [commandOptions]
 ran yarn rmadmin to get usage:
 RMAdmin
 Usage: java RMAdmin
[-refreshQueues]
[-refreshNodes]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshAdminAcls]
[-refreshServiceAcl]
[-help [cmd]]
 Generic options supported are
 -conf configuration file specify an application configuration file
 -D property=valueuse value for given property
 -fs local|namenode:port  specify a namenode
 -jt local|jobtracker:portspecify a job tracker
 -files comma separated list of filesspecify comma separated files to be 
 copied to the map reduce cluster
 -libjars comma separated list of jarsspecify comma separated jar files 
 to include in the classpath.
 -archives comma separated list of archivesspecify comma separated 
 archives to be unarchived on the compute machines.
 The general command line syntax is
 bin/hadoop command [genericOptions] [commandOptions]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart

2013-04-23 Thread Jian He (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639638#comment-13639638
 ] 

Jian He commented on YARN-581:
--

bq. I remember we discussed this on some other ticket too? Why not stop RM1 
before starting RM2 in the test-case?
Because its incorrect to stop rm1 first. If rm1.stop(), the store dispatcher is 
stopped, its not able to process remove event any more, app1 still remain. I 
also comment this on YARN-594

 Test and verify that app delegation tokens are restored after RM restart
 

 Key: YARN-581
 URL: https://issues.apache.org/jira/browse/YARN-581
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
 Attachments: YARN-581.1.patch


 The code already saves the delegation tokens in AppSubmissionContext. Upon 
 restart the AppSubmissionContext is used to submit the application again and 
 so restores the delegation tokens. This jira tracks testing and verifying 
 this functionality in a secure setup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-422) Add AM-NM client library

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639639#comment-13639639
 ] 

Vinod Kumar Vavilapalli commented on YARN-422:
--

Comments on your proposal
 - AMNMClient is good.
 - We should have only one instance per AM which talks to *all* NMs. Just like 
ContainerLauncherImpl in MR App. Clearly, the interface you proposed is already 
accommodating for that.
 - All the APIs will be blocking? It isn't clear. Today's MR App's 
ContainerLauncher is non-blocking and it has call-backs/signals as to what 
happened with a container launch/stop - succeeded/failed. I like making it 
non-blocking as AMs more commonly should want to kick off the launch and go 
away. A blocking API could be implemented on top of the non-blocking API if 
need be.
 - I think we should change AMLauncher also to use this, but we can scope it 
into a separate ticket depending on this patch's size

What do others think?

 Add AM-NM client library
 

 Key: YARN-422
 URL: https://issues.apache.org/jira/browse/YARN-422
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
 Attachments: proposal_v1.pdf


 Create a simple wrapper over the AM-NM container protocol to provide hide the 
 details of the protocol implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639640#comment-13639640
 ] 

Vinod Kumar Vavilapalli commented on YARN-561:
--

bq. there are also checks in the MRAppMaster which pretty much validate 
everything that the NM is supposed to be setting in the env. ( look for callers 
of MRAppMaster#validateInputParam ). This can be probably addressed separately 
in any case.
Yeah, let's do those separately.

The latest patch looks good, +1, checking it in.

Xuan, can you please file all the follow up tickets? Thanks.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, 
 YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, 
 YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Xuan Gong (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639643#comment-13639643
 ] 

Xuan Gong commented on YARN-561:


Yes, I will do that.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, 
 YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, 
 YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-562) NM should reject containers allocated by previous RM

2013-04-23 Thread Jian He (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-562:
-

Attachment: YARN-562.8.patch

New patch does:
1. Add test to verify new container requests are blocked when NM start from 
scratch until it registers with RM.
2. Add test to verify RMIdentifier is set in RegisterNodeManagerResponse
3. Add test to verify RMIdentifier is set on allocated containers.
4. Address above comments

 NM should reject containers allocated by previous RM
 

 Key: YARN-562
 URL: https://issues.apache.org/jira/browse/YARN-562
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, 
 YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, 
 YARN-562.8.patch


 Its possible that after RM shutdown, before AM goes down,AM still call 
 startContainer on NM with containers allocated by previous RM. When RM comes 
 back, NM doesn't know whether this container launch request comes from 
 previous RM or the current RM. we should reject containers allocated by 
 previous RM 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (YARN-601) Refactoring the code which computes the user file cache and user application file cache paths

2013-04-23 Thread Omkar Vinit Joshi (JIRA)
Omkar Vinit Joshi created YARN-601:
--

 Summary: Refactoring the code which computes the user file cache 
and user application file cache paths
 Key: YARN-601
 URL: https://issues.apache.org/jira/browse/YARN-601
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Omkar Vinit Joshi
Priority: Minor


At present at multiple places user local cache file path and user application 
file path are getting calculated.
It is better to expose them as a single static utility method and reuse them 
everywhere else.
Locations :
* ContainerLaunch
* DefaultContainerExecutor: this already has some methods like this
* ResourceLocalizationService: getUserCacheFilePath getAppFileCachePath
* ContainerLocalizer
* ShuffleHandler.Shuffle
* TestContainerLocalizer, TestContainerManager, TestDefaultContainerExecutor 
and TestResourceLocalizationService.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-562) NM should reject containers allocated by previous RM

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639659#comment-13639659
 ] 

Hadoop QA commented on YARN-562:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580136/YARN-562.8.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 17 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/809//console

This message is automatically generated.

 NM should reject containers allocated by previous RM
 

 Key: YARN-562
 URL: https://issues.apache.org/jira/browse/YARN-562
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, 
 YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, 
 YARN-562.8.patch


 Its possible that after RM shutdown, before AM goes down,AM still call 
 startContainer on NM with containers allocated by previous RM. When RM comes 
 back, NM doesn't know whether this container launch request comes from 
 previous RM or the current RM. we should reject containers allocated by 
 previous RM 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-422) Add AM-NM client library

2013-04-23 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639660#comment-13639660
 ] 

Bikas Saha commented on YARN-422:
-

I was hoping that the final AMNMClient library would be a one stop library for 
AM's to communicate with all the NM's in the cluster. The document outlines a 
wrapper around the NMProtocol for talking to 1 NM. This wrapper is useful by 
itself because it hides the PBImpl etc implementation details of the protocol 
itself, similar to how YarnClient started off as. AMRMClient does a little bit 
more than simply wrapping the Java protocol.
So as a first step the above design for wrapping the boiler plate code of the 
NM Protocol into a utility library wrapper sounds good. I think the method 
calls are all blocking since the RPC itself is blocking and this is a simple 
wrapper around the RPC. Maybe we can call be NMClient.

Once this is done we can look at creating an AMNMClient library that will 
provide a single object that an AM can use to manage its containers. This may 
have its own internal threading etc which will allow the AM to make 
non-blocking calls. Lets figure that out in a subsequent jira and use this one 
to just write the wrapper NMCLient that you have proposed.

 Add AM-NM client library
 

 Key: YARN-422
 URL: https://issues.apache.org/jira/browse/YARN-422
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
 Attachments: proposal_v1.pdf


 Create a simple wrapper over the AM-NM container protocol to provide hide the 
 details of the protocol implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-581) Test and verify that app delegation tokens are restored after RM restart

2013-04-23 Thread Jian He (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-581:
-

Attachment: YARN-581.2.patch

New patch fixed above comments

 Test and verify that app delegation tokens are restored after RM restart
 

 Key: YARN-581
 URL: https://issues.apache.org/jira/browse/YARN-581
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
 Attachments: YARN-581.1.patch, YARN-581.2.patch


 The code already saves the delegation tokens in AppSubmissionContext. Upon 
 restart the AppSubmissionContext is used to submit the application again and 
 so restores the delegation tokens. This jira tracks testing and verifying 
 this functionality in a secure setup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639673#comment-13639673
 ] 

Vinod Kumar Vavilapalli commented on YARN-561:
--

Xuan, we need a separate patch for branch-2, the trunk patch isn't applying. 
Can you please generate one, run all the tests on branch-2 applying that patch? 
We can't do that on Jenkins, as it isn't built for running tests on branches. 
Tx.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, YARN-561.1.patch, 
 YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, 
 YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (YARN-602) NodeManager should mandatorily set some Environment variables into every containers that it launches

2013-04-23 Thread Xuan Gong (JIRA)
Xuan Gong created YARN-602:
--

 Summary: NodeManager should mandatorily set some Environment 
variables into every containers that it launches
 Key: YARN-602
 URL: https://issues.apache.org/jira/browse/YARN-602
 Project: Hadoop YARN
  Issue Type: Bug
 Environment: NodeManager should mandatorily set some Environment 
variables into every containers that it launches, such as Environment.user, 
Environment.pwd. If both users and NodeManager set those variables, the value 
set by NM should be used 
Reporter: Xuan Gong




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (YARN-603) Create testcase to validate Environment.MALLOC_ARENA_MAX

2013-04-23 Thread Xuan Gong (JIRA)
Xuan Gong created YARN-603:
--

 Summary: Create testcase to validate Environment.MALLOC_ARENA_MAX
 Key: YARN-603
 URL: https://issues.apache.org/jira/browse/YARN-603
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong


The current test to validate Environment.MALLOC_ARENA_MAX isn't sufficient. We 
need validate YarnConfiguration.NM_ADMIN_USER_ENV, too. 
And YARN-561 removed testing of Environment.MALLOC_ARENA_MAX, we need to create 
new test case to test it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639688#comment-13639688
 ] 

Hadoop QA commented on YARN-581:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580139/YARN-581.2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/810//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/810//console

This message is automatically generated.

 Test and verify that app delegation tokens are restored after RM restart
 

 Key: YARN-581
 URL: https://issues.apache.org/jira/browse/YARN-581
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
 Attachments: YARN-581.1.patch, YARN-581.2.patch


 The code already saves the delegation tokens in AppSubmissionContext. Upon 
 restart the AppSubmissionContext is used to submit the application again and 
 so restores the delegation tokens. This jira tracks testing and verifying 
 this functionality in a secure setup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639748#comment-13639748
 ] 

Vinod Kumar Vavilapalli commented on YARN-581:
--

Latest patch looks good, checking it in.

 Test and verify that app delegation tokens are restored after RM restart
 

 Key: YARN-581
 URL: https://issues.apache.org/jira/browse/YARN-581
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
 Attachments: YARN-581.1.patch, YARN-581.2.patch


 The code already saves the delegation tokens in AppSubmissionContext. Upon 
 restart the AppSubmissionContext is used to submit the application again and 
 so restores the delegation tokens. This jira tracks testing and verifying 
 this functionality in a secure setup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-581) Test and verify that app delegation tokens are restored after RM restart

2013-04-23 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639761#comment-13639761
 ] 

Hudson commented on YARN-581:
-

Integrated in Hadoop-trunk-Commit #3652 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3652/])
YARN-581. Added a test to verify that app delegation tokens are restored 
after RM restart. Contributed by Jian He. (Revision 1471187)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1471187
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java


 Test and verify that app delegation tokens are restored after RM restart
 

 Key: YARN-581
 URL: https://issues.apache.org/jira/browse/YARN-581
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Jian He
 Fix For: 2.0.5-beta

 Attachments: YARN-581.1.patch, YARN-581.2.patch


 The code already saves the delegation tokens in AppSubmissionContext. Upon 
 restart the AppSubmissionContext is used to submit the application again and 
 so restores the delegation tokens. This jira tracks testing and verifying 
 this functionality in a secure setup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-422) Add AM-NM client library

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639767#comment-13639767
 ] 

Vinod Kumar Vavilapalli commented on YARN-422:
--

I don't see any value in a wrapper around communication to a single NM. It does 
hide-away the records, but what is useful and important is a library which 
takes in a container given by RM and launches in whatever node it is allocated 
on. In that sense, this ContainerLauncher library is what should be the 
primary end-point for AM writers. If one wants to launch containers on a single 
node, the same library should be enough.

And yeah, the more I think about, non-blocking APIs seem like must-have.

 Add AM-NM client library
 

 Key: YARN-422
 URL: https://issues.apache.org/jira/browse/YARN-422
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
 Attachments: proposal_v1.pdf


 Create a simple wrapper over the AM-NM container protocol to provide hide the 
 details of the protocol implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-562) NM should reject containers allocated by previous RM

2013-04-23 Thread Jian He (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-562:
-

Attachment: YARN-562.9.patch

fixed build failure..

 NM should reject containers allocated by previous RM
 

 Key: YARN-562
 URL: https://issues.apache.org/jira/browse/YARN-562
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, 
 YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, 
 YARN-562.8.patch, YARN-562.9.patch


 Its possible that after RM shutdown, before AM goes down,AM still call 
 startContainer on NM with containers allocated by previous RM. When RM comes 
 back, NM doesn't know whether this container launch request comes from 
 previous RM or the current RM. we should reject containers allocated by 
 previous RM 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-561:
---

Attachment: YARN-561.12.branch_2.patch

Create patch for branch_2 only

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, 
 YARN-561.12.branch_2.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639794#comment-13639794
 ] 

Hadoop QA commented on YARN-561:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12580178/YARN-561.12.branch_2.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/812//console

This message is automatically generated.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, 
 YARN-561.12.branch_2.patch, YARN-561.1.patch, YARN-561.2.patch, 
 YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, YARN-561.6.patch, 
 YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-422) Add AM-NM client library

2013-04-23 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639834#comment-13639834
 ] 

Bikas Saha commented on YARN-422:
-

Sounds good to me to skip the first wrapper step if there are no consumers for 
it. This would still be an NMClient library since potentially the RM and AM can 
both use it.

 Add AM-NM client library
 

 Key: YARN-422
 URL: https://issues.apache.org/jira/browse/YARN-422
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
 Attachments: proposal_v1.pdf


 Create a simple wrapper over the AM-NM container protocol to provide hide the 
 details of the protocol implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-561:
---

Attachment: YARN-561.13.branch-2.patch

1.Create new patch for branch-2
2.Pass all Yarn test cases locally

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Attachments: YARN-561.10.patch, YARN-561.11.patch, 
 YARN-561.12.branch_2.patch, YARN-561.13.branch-2.patch, YARN-561.1.patch, 
 YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, 
 YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-562) NM should reject containers allocated by previous RM

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639844#comment-13639844
 ] 

Hadoop QA commented on YARN-562:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580175/YARN-562.9.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 17 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/811//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/811//console

This message is automatically generated.

 NM should reject containers allocated by previous RM
 

 Key: YARN-562
 URL: https://issues.apache.org/jira/browse/YARN-562
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, 
 YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, 
 YARN-562.8.patch, YARN-562.9.patch


 Its possible that after RM shutdown, before AM goes down,AM still call 
 startContainer on NM with containers allocated by previous RM. When RM comes 
 back, NM doesn't know whether this container launch request comes from 
 previous RM or the current RM. we should reject containers allocated by 
 previous RM 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-599) Refactoring submitApplication in ClientRMService and RMAppManager

2013-04-23 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639848#comment-13639848
 ] 

Zhijie Shen commented on YARN-599:
--

In YARN-599.1.patch, there're the following changes:

1. ClientRMService#submitApplication calls RMAppManager#submitApplication 
directly. APP_SUBMIT event is removed at all. RMAppManager#submitApplication 
throws YarnRemoteException.

2. Move getCurrentUser and validateResourceRequest from 
ClientRMService#submitApplication to RMAppManager#submitApplication. Move 
getQueue and getApplicationName from RMAppManager#submitApplication to 
ClientRMService#submitApplication. Adjust the test cases in TestClientRMService 
and TestAppManger accordingly.

3. Refactor try-catch block in RMAppManager#submitApplication to avoid sending 
APP_REJECTED event to existing app in rmContext given duplicate applicateId.

4. Refactor TestAppManger to extract common part of the tests and push them to 
setup().

 Refactoring submitApplication in ClientRMService and RMAppManager
 -

 Key: YARN-599
 URL: https://issues.apache.org/jira/browse/YARN-599
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-599.1.patch


 Currently, ClientRMService#submitApplication call RMAppManager#handle, and 
 consequently call RMAppMangager#submitApplication directly, though the code 
 looks like scheduling an APP_SUBMIT event.
 In addition, the validation code before creating an RMApp instance is not 
 well organized. Ideally, the dynamic validation, which depends on the RM's 
 configuration, should be put in RMAppMangager#submitApplication. 
 RMAppMangager#submitApplication is called by 
 ClientRMService#submitApplication and RMAppMangager#recover. Since the 
 configuration may be changed after RM restarts, the validation needs to be 
 done again even in recovery mode. Therefore, resource request validation, 
 which based on min/max resource limits, should be moved from 
 ClientRMService#submitApplication to RMAppMangager#submitApplication. On the 
 other hand, the static validation, which is independent of the RM's 
 configuration should be put in ClientRMService#submitApplication, because it 
 is only need to be done once during the first submission.
 Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. 
 RMAppMangager#submitApplication has a flaw is not synchronized. If two 
 application submissions with the same application ID enter the function, and 
 one progresses to the completion of RMApp instantiation, and the other 
 progresses the completion of putting the RMApp instance into rmContext, the 
 slower submission will cause an exception due to the duplicate application 
 ID. However, the exception will cause the RMApp instance already in rmContext 
 (belongs to the faster submission) being rejected with the current code flow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Resolved] (YARN-561) Nodemanager should set some key information into the environment of every container that it launches.

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-561.
--

   Resolution: Fixed
Fix Version/s: 2.0.5-beta
 Hadoop Flags: Incompatible change,Reviewed

branch-2 patch looks good too. +1.

Committed this to trunk and branch-2. Thanks Xuan.

Marking this as incompatible, as it is renaming the LOCAL_DIRS env constant.

 Nodemanager should set some key information into the environment of every 
 container that it launches.
 -

 Key: YARN-561
 URL: https://issues.apache.org/jira/browse/YARN-561
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Xuan Gong
  Labels: usability
 Fix For: 2.0.5-beta

 Attachments: YARN-561.10.patch, YARN-561.11.patch, 
 YARN-561.12.branch_2.patch, YARN-561.13.branch-2.patch, YARN-561.1.patch, 
 YARN-561.2.patch, YARN-561.3.patch, YARN-561.4.patch, YARN-561.5.patch, 
 YARN-561.6.patch, YARN-561.7.patch, YARN-561.8.patch, YARN-561.9.patch


 Information such as containerId, nodemanager hostname, nodemanager port is 
 not set in the environment when any container is launched. 
 For an AM, the RM does all of this for it but for a container launched by an 
 application, all of the above need to be set by the ApplicationMaster. 
 At the minimum, container id would be a useful piece of information. If the 
 container wishes to talk to its local NM, the nodemanager related information 
 would also come in handy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-603) Create a testcase to validate Environment.MALLOC_ARENA_MAX

2013-04-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-603:
---

Summary: Create a testcase to validate Environment.MALLOC_ARENA_MAX  (was: 
Create testcase to validate Environment.MALLOC_ARENA_MAX)

 Create a testcase to validate Environment.MALLOC_ARENA_MAX
 --

 Key: YARN-603
 URL: https://issues.apache.org/jira/browse/YARN-603
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong

 The current test to validate Environment.MALLOC_ARENA_MAX isn't sufficient. 
 We need validate YarnConfiguration.NM_ADMIN_USER_ENV, too. 
 And YARN-561 removed testing of Environment.MALLOC_ARENA_MAX, we need to 
 create new test case to test it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (YARN-604) The checks in the MRAppMaster should validate everything that the NM is supposed to be setting in the env

2013-04-23 Thread Xuan Gong (JIRA)
Xuan Gong created YARN-604:
--

 Summary: The checks in the MRAppMaster should validate everything 
that the NM is supposed to be setting in the env
 Key: YARN-604
 URL: https://issues.apache.org/jira/browse/YARN-604
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-604) The checkers in the MRAppMaster should validate everything that the NM is supposed to be setting in the env

2013-04-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-604:
---

Summary: The checkers in the MRAppMaster should validate everything that 
the NM is supposed to be setting in the env  (was: The checks in the 
MRAppMaster should validate everything that the NM is supposed to be setting in 
the env)

 The checkers in the MRAppMaster should validate everything that the NM is 
 supposed to be setting in the env
 ---

 Key: YARN-604
 URL: https://issues.apache.org/jira/browse/YARN-604
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-595) Refactor fair scheduler to use common Resources

2013-04-23 Thread Sandy Ryza (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated YARN-595:


Attachment: YARN-595-1.patch

 Refactor fair scheduler to use common Resources
 ---

 Key: YARN-595
 URL: https://issues.apache.org/jira/browse/YARN-595
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-595-1.patch, YARN-595.patch, YARN-595.patch


 resourcemanager.fair and resourcemanager.resources have two copies of 
 basically the same code for operations on Resource objects

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-595) Refactor fair scheduler to use common Resources

2013-04-23 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639901#comment-13639901
 ] 

Sandy Ryza commented on YARN-595:
-

Updated patch addresses Alejandro's comments

 Refactor fair scheduler to use common Resources
 ---

 Key: YARN-595
 URL: https://issues.apache.org/jira/browse/YARN-595
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-595-1.patch, YARN-595.patch, YARN-595.patch


 resourcemanager.fair and resourcemanager.resources have two copies of 
 basically the same code for operations on Resource objects

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-604) The checkers in the MRAppMaster should validate everything that the NM is supposed to be setting in the env

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639911#comment-13639911
 ] 

Vinod Kumar Vavilapalli commented on YARN-604:
--

No, wait. I thought we don't need to validate them anymore as NM itself is 
setting it. Isn't that the case?

 The checkers in the MRAppMaster should validate everything that the NM is 
 supposed to be setting in the env
 ---

 Key: YARN-604
 URL: https://issues.apache.org/jira/browse/YARN-604
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-562) NM should reject containers allocated by previous RM

2013-04-23 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639914#comment-13639914
 ] 

Vinod Kumar Vavilapalli commented on YARN-562:
--

Looking at the patch for final review/commit.

 NM should reject containers allocated by previous RM
 

 Key: YARN-562
 URL: https://issues.apache.org/jira/browse/YARN-562
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-562.1.patch, YARN-562.2.patch, YARN-562.3.patch, 
 YARN-562.4.patch, YARN-562.5.patch, YARN-562.6.patch, YARN-562.7.patch, 
 YARN-562.8.patch, YARN-562.9.patch


 Its possible that after RM shutdown, before AM goes down,AM still call 
 startContainer on NM with containers allocated by previous RM. When RM comes 
 back, NM doesn't know whether this container launch request comes from 
 previous RM or the current RM. we should reject containers allocated by 
 previous RM 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-602) NodeManager should mandatorily set some Environment variables into every containers that it launches

2013-04-23 Thread Hitesh Shah (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated YARN-602:
-

Description: NodeManager should mandatorily set some Environment variables 
into every containers that it launches, such as Environment.user, 
Environment.pwd. If both users and NodeManager set those variables, the value 
set by NM should be used 

 NodeManager should mandatorily set some Environment variables into every 
 containers that it launches
 

 Key: YARN-602
 URL: https://issues.apache.org/jira/browse/YARN-602
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong

 NodeManager should mandatorily set some Environment variables into every 
 containers that it launches, such as Environment.user, Environment.pwd. If 
 both users and NodeManager set those variables, the value set by NM should be 
 used 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-602) NodeManager should mandatorily set some Environment variables into every containers that it launches

2013-04-23 Thread Hitesh Shah (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated YARN-602:
-

Environment: (was: NodeManager should mandatorily set some Environment 
variables into every containers that it launches, such as Environment.user, 
Environment.pwd. If both users and NodeManager set those variables, the value 
set by NM should be used )

 NodeManager should mandatorily set some Environment variables into every 
 containers that it launches
 

 Key: YARN-602
 URL: https://issues.apache.org/jira/browse/YARN-602
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-595) Refactor fair scheduler to use common Resources

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639950#comment-13639950
 ] 

Hadoop QA commented on YARN-595:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580198/YARN-595-1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/813//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/813//console

This message is automatically generated.

 Refactor fair scheduler to use common Resources
 ---

 Key: YARN-595
 URL: https://issues.apache.org/jira/browse/YARN-595
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-595-1.patch, YARN-595.patch, YARN-595.patch


 resourcemanager.fair and resourcemanager.resources have two copies of 
 basically the same code for operations on Resource objects

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Assigned] (YARN-577) ApplicationReport does not provide progress value of application

2013-04-23 Thread Hitesh Shah (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah reassigned YARN-577:


Assignee: Hitesh Shah  (was: Vinod Kumar Vavilapalli)

 ApplicationReport does not provide progress value of application
 

 Key: YARN-577
 URL: https://issues.apache.org/jira/browse/YARN-577
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Hitesh Shah

 An application sends its progress % to the RM via AllocateRequest. This 
 should be able to be retrieved by a client via the ApplicationReport.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-556) RM Restart phase 2 - Design for work preserving restart

2013-04-23 Thread Bikas Saha (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated YARN-556:


Labels: gsoc2013  (was: )

 RM Restart phase 2 - Design for work preserving restart
 ---

 Key: YARN-556
 URL: https://issues.apache.org/jira/browse/YARN-556
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: gsoc2013

 The basic idea is already documented on YARN-128. This will describe further 
 details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-556) RM Restart phase 2 - Design for work preserving restart

2013-04-23 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640012#comment-13640012
 ] 

Bikas Saha commented on YARN-556:
-

Adding brief description of proposal from YARN-128 design document for Google 
Summer of Code.
{noformat}
Work preserving restart - RM will have to make the RMAppAttempt state machine 
enter 
the Running state before starting the internal services. The RM can obtain 
information 
about all running containers from the NM’s when the NM’s heartbeat with it. 
This 
information can be used to repopulate the allocation state of scheduler. When 
the 
running AM’s heartbeat with RM then the RM can ask them to resend their 
container 
requests so that the RM can repopulate all the pending requests. Repopulating 
the 
running container and pending container information completes all the data 
needed by 
the RM to start normal operations.
{noformat}

 RM Restart phase 2 - Design for work preserving restart
 ---

 Key: YARN-556
 URL: https://issues.apache.org/jira/browse/YARN-556
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: gsoc2013

 The basic idea is already documented on YARN-128. This will describe further 
 details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (YARN-605) Failing unit test in TestNMWebServices when using git for source control

2013-04-23 Thread Hitesh Shah (JIRA)
Hitesh Shah created YARN-605:


 Summary: Failing unit test in TestNMWebServices when using git for 
source control 
 Key: YARN-605
 URL: https://issues.apache.org/jira/browse/YARN-605
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah


Failed tests:   
testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): 
hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 
3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
f89f5c9b9c9d44cf3be5c2686f2d789
  
testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 
3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
f89f5c9b9c9d44cf3be5c2686f2d789
  
testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 
3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
f89f5c9b9c9d44cf3be5c2686f2d789
  
testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 
3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
f89f5c9b9c9d44cf3be5c2686f2d789
  
testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 
3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
f89f5c9b9c9d44cf3be5c2686f2d789
  
testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 
3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
f89f5c9b9c9d44cf3be5c2686f2d789
  
testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 expected: 
3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
f89f5c9b9c9d44cf3be5c2686f2d789

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-605) Failing unit test in TestNMWebServices when using git for source control

2013-04-23 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640048#comment-13640048
 ] 

Hitesh Shah commented on YARN-605:
--

This seems to fix it:

{code}
diff --git 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-
index d82771b..b1cff2c 100644
--- 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java
+++ 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java
@@ -76,7 +76,7 @@ public static String getXmlAttrString(Element element, String 
name) {
   public static void checkStringMatch(String print, String expected, String 
got) {
 assertTrue(
 print +  doesn't match, got:  + got +  expected:  + expected,
-got.matches(expected));
+got.equals(expected));
   }
 
   public static void checkStringContains(String print, String expected, String 
got) {
{code}

[~tgraves] Any comments on whether matches() is needed? I am guessing something 
in the strings ( maybe a '-' ) is causing problems when using a regex match.

 Failing unit test in TestNMWebServices when using git for source control 
 -

 Key: YARN-605
 URL: https://issues.apache.org/jira/browse/YARN-605
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Failed tests:   
 testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): 
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh 

[jira] [Assigned] (YARN-605) Failing unit test in TestNMWebServices when using git for source control

2013-04-23 Thread Hitesh Shah (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah reassigned YARN-605:


Assignee: Hitesh Shah

 Failing unit test in TestNMWebServices when using git for source control 
 -

 Key: YARN-605
 URL: https://issues.apache.org/jira/browse/YARN-605
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah
Assignee: Hitesh Shah
 Attachments: YARN-605.1.patch


 Failed tests:   
 testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): 
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-605) Failing unit test in TestNMWebServices when using git for source control

2013-04-23 Thread Hitesh Shah (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated YARN-605:
-

Attachment: YARN-605.1.patch

 Failing unit test in TestNMWebServices when using git for source control 
 -

 Key: YARN-605
 URL: https://issues.apache.org/jira/browse/YARN-605
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah
Assignee: Hitesh Shah
 Attachments: YARN-605.1.patch


 Failed tests:   
 testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): 
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testSingleNodesXML(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Updated] (YARN-577) ApplicationReport does not provide progress value of application

2013-04-23 Thread Hitesh Shah (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated YARN-577:
-

Attachment: YARN-577.1.patch

 ApplicationReport does not provide progress value of application
 

 Key: YARN-577
 URL: https://issues.apache.org/jira/browse/YARN-577
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Hitesh Shah
 Attachments: YARN-577.1.patch


 An application sends its progress % to the RM via AllocateRequest. This 
 should be able to be retrieved by a client via the ApplicationReport.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-556) RM Restart phase 2 - Design for work preserving restart

2013-04-23 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640108#comment-13640108
 ] 

Junping Du commented on YARN-556:
-

Hi Bikas, beside resending resource request through AM-RM hearbeat, do we think 
some other way like: putting resource requests to ZK store?

 RM Restart phase 2 - Design for work preserving restart
 ---

 Key: YARN-556
 URL: https://issues.apache.org/jira/browse/YARN-556
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Bikas Saha
Assignee: Bikas Saha
  Labels: gsoc2013

 The basic idea is already documented on YARN-128. This will describe further 
 details.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-577) ApplicationReport does not provide progress value of application

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640118#comment-13640118
 ] 

Hadoop QA commented on YARN-577:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580238/YARN-577.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/815//console

This message is automatically generated.

 ApplicationReport does not provide progress value of application
 

 Key: YARN-577
 URL: https://issues.apache.org/jira/browse/YARN-577
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Hitesh Shah
Assignee: Hitesh Shah
 Attachments: YARN-577.1.patch


 An application sends its progress % to the RM via AllocateRequest. This 
 should be able to be retrieved by a client via the ApplicationReport.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-605) Failing unit test in TestNMWebServices when using git for source control

2013-04-23 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640123#comment-13640123
 ] 

Hadoop QA commented on YARN-605:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12580234/YARN-605.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/814//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/814//console

This message is automatically generated.

 Failing unit test in TestNMWebServices when using git for source control 
 -

 Key: YARN-605
 URL: https://issues.apache.org/jira/browse/YARN-605
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah
Assignee: Hitesh Shah
 Attachments: YARN-605.1.patch


 Failed tests:   
 testNode(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices): 
 hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfo(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoSlash(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track) by Hitesh source checksum 
 f89f5c9b9c9d44cf3be5c2686f2d789
   
 testNodeInfoDefault(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices):
  hadoopBuildVersion doesn't match, got: 3.0.0-SNAPSHOT from 
 fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, origin/trunk, origin/HEAD, 
 mrx-track) by Hitesh source checksum f89f5c9b9c9d44cf3be5c2686f2d789 
 expected: 3.0.0-SNAPSHOT from fddcdcfb3cfe7dcc4f77c1ac953dd2cc0a890c62 (HEAD, 
 origin/trunk, origin/HEAD, mrx-track)