date:20140211


[ 
https://issues.apache.org/jira/browse/YARN-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897741#comment-13897741
 ] 

Hudson commented on YARN-1637:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/478/])
YARN-1637. Implemented a client library for Java users to post timeline 
entities and events. Contributed by Zhijie Shen. (zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566752)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java


 Implement a client library for java users to post entities+events
 -

 Key: YARN-1637
 URL: https://issues.apache.org/jira/browse/YARN-1637
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1637.1.patch, YARN-1637.2.patch, YARN-1637.3.patch, 
 YARN-1637.4.patch, YARN-1637.5.patch, YARN-1637.6.patch, YARN-1637.7.patch, 
 YARN-1637.8.patch


 This is a wrapper around the web-service to facilitate easy posting of 
 entity+event data to the time-line server.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1698) Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default


[ 
https://issues.apache.org/jira/browse/YARN-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897743#comment-13897743
 ] 

Hudson commented on YARN-1698:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/478/])
YARN-1698. Fixed default TimelineStore in code to match what is documented in 
yarn-default.xml. Contributed by Zhijie Shen. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566937)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java


 Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore 
 as default
 --

 Key: YARN-1698
 URL: https://issues.apache.org/jira/browse/YARN-1698
 Project: Hadoop YARN
  Issue Type: Sub-task
Affects Versions: 2.4.0
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1698.1.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1459) RM services should depend on ConfigurationProvider during startup too


[ 
https://issues.apache.org/jira/browse/YARN-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897747#comment-13897747
 ] 

Hudson commented on YARN-1459:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/478/])
YARN-1459. Changed ResourceManager to depend its service initialization on the 
configuration-provider mechanism during startup too. Contributed by Xuan Gong. 
(vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566791)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProviderFactory.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/FileSystemBasedConfigurationProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/LocalConfigurationProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java


 RM services should depend on ConfigurationProvider during startup too
 -

 Key: YARN-1459
 URL: https://issues.apache.org/jira/browse/YARN-1459
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.2.0
Reporter: Karthik Kambatla
Assignee: Xuan Gong
 Fix For: 2.4.0

 Attachments: YARN-1459.1.patch, YARN-1459.2.patch, YARN-1459.3.patch, 
 YARN-1459.4.patch, YARN-1459.4.patch


 YARN-1667, YARN-1668, YARN-1669 already changed RM to depend on a 
 configuration provider so as to be able to refresh many configuration files 
 across RM fail-over. The dependency on the configuration-provider by the RM 
 should happen at its boot up time too.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1706) Create an utility function to dump timeline records to json


[ 
https://issues.apache.org/jira/browse/YARN-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897745#comment-13897745
 ] 

Hudson commented on YARN-1706:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/478/])
YARN-1706. Created an utility method to dump timeline records to JSON strings. 
Contributed by Zhijie Shen. (zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566982)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/TimelineUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java


 Create an utility function to dump timeline records to json 
 

 Key: YARN-1706
 URL: https://issues.apache.org/jira/browse/YARN-1706
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1706.1.patch, YARN-1706.2.patch, YARN-1706.3.patch, 
 YARN-1706.4.patch


 For verification and log purpose



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (YARN-1702) Expose kill app functionality as part of RM web services

2014-02-11 Thread Varun Vasudev (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev reassigned YARN-1702:
---

Assignee: Varun Vasudev

 Expose kill app functionality as part of RM web services
 

 Key: YARN-1702
 URL: https://issues.apache.org/jira/browse/YARN-1702
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev

 Expose functionality to kill an app via the ResourceManager web services API.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1637) Implement a client library for java users to post entities+events


[ 
https://issues.apache.org/jira/browse/YARN-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897840#comment-13897840
 ] 

Hudson commented on YARN-1637:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/])
YARN-1637. Implemented a client library for Java users to post timeline 
entities and events. Contributed by Zhijie Shen. (zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566752)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java


 Implement a client library for java users to post entities+events
 -

 Key: YARN-1637
 URL: https://issues.apache.org/jira/browse/YARN-1637
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1637.1.patch, YARN-1637.2.patch, YARN-1637.3.patch, 
 YARN-1637.4.patch, YARN-1637.5.patch, YARN-1637.6.patch, YARN-1637.7.patch, 
 YARN-1637.8.patch


 This is a wrapper around the web-service to facilitate easy posting of 
 entity+event data to the time-line server.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1459) RM services should depend on ConfigurationProvider during startup too


[ 
https://issues.apache.org/jira/browse/YARN-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897846#comment-13897846
 ] 

Hudson commented on YARN-1459:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/])
YARN-1459. Changed ResourceManager to depend its service initialization on the 
configuration-provider mechanism during startup too. Contributed by Xuan Gong. 
(vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566791)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProviderFactory.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/FileSystemBasedConfigurationProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/LocalConfigurationProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java


 RM services should depend on ConfigurationProvider during startup too
 -

 Key: YARN-1459
 URL: https://issues.apache.org/jira/browse/YARN-1459
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.2.0
Reporter: Karthik Kambatla
Assignee: Xuan Gong
 Fix For: 2.4.0

 Attachments: YARN-1459.1.patch, YARN-1459.2.patch, YARN-1459.3.patch, 
 YARN-1459.4.patch, YARN-1459.4.patch


 YARN-1667, YARN-1668, YARN-1669 already changed RM to depend on a 
 configuration provider so as to be able to refresh many configuration files 
 across RM fail-over. The dependency on the configuration-provider by the RM 
 should happen at its boot up time too.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1706) Create an utility function to dump timeline records to json


[ 
https://issues.apache.org/jira/browse/YARN-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897844#comment-13897844
 ] 

Hudson commented on YARN-1706:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/])
YARN-1706. Created an utility method to dump timeline records to JSON strings. 
Contributed by Zhijie Shen. (zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566982)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/TimelineUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java


 Create an utility function to dump timeline records to json 
 

 Key: YARN-1706
 URL: https://issues.apache.org/jira/browse/YARN-1706
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1706.1.patch, YARN-1706.2.patch, YARN-1706.3.patch, 
 YARN-1706.4.patch


 For verification and log purpose



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1698) Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default


[ 
https://issues.apache.org/jira/browse/YARN-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897842#comment-13897842
 ] 

Hudson commented on YARN-1698:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/])
YARN-1698. Fixed default TimelineStore in code to match what is documented in 
yarn-default.xml. Contributed by Zhijie Shen. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566937)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java


 Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore 
 as default
 --

 Key: YARN-1698
 URL: https://issues.apache.org/jira/browse/YARN-1698
 Project: Hadoop YARN
  Issue Type: Sub-task
Affects Versions: 2.4.0
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1698.1.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1706) Create an utility function to dump timeline records to json


[ 
https://issues.apache.org/jira/browse/YARN-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897889#comment-13897889
 ] 

Hudson commented on YARN-1706:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/])
YARN-1706. Created an utility method to dump timeline records to JSON strings. 
Contributed by Zhijie Shen. (zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566982)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/TimelineUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java


 Create an utility function to dump timeline records to json 
 

 Key: YARN-1706
 URL: https://issues.apache.org/jira/browse/YARN-1706
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1706.1.patch, YARN-1706.2.patch, YARN-1706.3.patch, 
 YARN-1706.4.patch


 For verification and log purpose



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1698) Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default


[ 
https://issues.apache.org/jira/browse/YARN-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897887#comment-13897887
 ] 

Hudson commented on YARN-1698:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/])
YARN-1698. Fixed default TimelineStore in code to match what is documented in 
yarn-default.xml. Contributed by Zhijie Shen. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566937)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java


 Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore 
 as default
 --

 Key: YARN-1698
 URL: https://issues.apache.org/jira/browse/YARN-1698
 Project: Hadoop YARN
  Issue Type: Sub-task
Affects Versions: 2.4.0
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1698.1.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1637) Implement a client library for java users to post entities+events


[ 
https://issues.apache.org/jira/browse/YARN-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897885#comment-13897885
 ] 

Hudson commented on YARN-1637:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/])
YARN-1637. Implemented a client library for Java users to post timeline 
entities and events. Contributed by Zhijie Shen. (zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566752)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java


 Implement a client library for java users to post entities+events
 -

 Key: YARN-1637
 URL: https://issues.apache.org/jira/browse/YARN-1637
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
 Fix For: 2.4.0

 Attachments: YARN-1637.1.patch, YARN-1637.2.patch, YARN-1637.3.patch, 
 YARN-1637.4.patch, YARN-1637.5.patch, YARN-1637.6.patch, YARN-1637.7.patch, 
 YARN-1637.8.patch


 This is a wrapper around the web-service to facilitate easy posting of 
 entity+event data to the time-line server.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1713) Implement getnewapplication as part of RM web service

2014-02-11 Thread Varun Vasudev (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-1713:


Attachment: yarn-1713.patch

 Implement getnewapplication as part of RM web service
 -

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1713) Implement getnewapplication as part of RM web service

2014-02-11 Thread Varun Vasudev (JIRA)

Varun Vasudev created YARN-1713:
---

 Summary: Implement getnewapplication as part of RM web service
 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: yarn-1713.patch





--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services

2014-02-11 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898052#comment-13898052
 ] 

Alejandro Abdelnur commented on YARN-1695:
--

err, [~vinodkv], isn't this a duplicate of the JIRA you opened back in DEC, 
YARN-1538?

 Implement the rest (writable APIs) of RM web-services
 -

 Key: YARN-1695
 URL: https://issues.apache.org/jira/browse/YARN-1695
 Project: Hadoop YARN
  Issue Type: New Feature
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli

 MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs 
 added there were only focused on obtaining information from the cluster. We 
 need to have the following REST APIs to finish the feature
  - Application submission/termination (Priority): This unblocks easy client 
 interaction with a YARN cluster
  - Application Client protocol: For resource scheduling by apps written in an 
 arbitrary language. Will have to think about throughput concerns
  - ContainerManagement Protocol: Again for arbitrary language apps.
 One important thing to note here is that we already have client libraries on 
 all the three protocols that do some some heavy-lifting. One part of the 
 effort is to figure out if they can be made any thinner and/or how 
 web-services will implement the same functionality.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (YARN-191) Enhancements to YARN for Windows Server and Windows Azure development and runtime environments

2014-02-11 Thread Chris Nauroth (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth resolved YARN-191.


Resolution: Fixed

We've completed the intended scope of this issue, so I'm resolving it.  Thank 
you to all contributors!

 Enhancements to YARN for Windows Server and Windows Azure development and 
 runtime environments
 --

 Key: YARN-191
 URL: https://issues.apache.org/jira/browse/YARN-191
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha

 This JIRA tracks the work that needs to be done on trunk to enable Hadoop to 
 run on Windows Server and Azure environments. This incorporates porting 
 relevant work from the similar effort on branch 1 tracked via HADOOP-8079.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services


[ 
https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898089#comment-13898089
 ] 

Vinod Kumar Vavilapalli commented on YARN-1695:
---

Ah, thanks. I did search for existing tickets but couldn't quite locate it 
using JIRA search. Closing one of these as a duplicate.

 Implement the rest (writable APIs) of RM web-services
 -

 Key: YARN-1695
 URL: https://issues.apache.org/jira/browse/YARN-1695
 Project: Hadoop YARN
  Issue Type: New Feature
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli

 MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs 
 added there were only focused on obtaining information from the cluster. We 
 need to have the following REST APIs to finish the feature
  - Application submission/termination (Priority): This unblocks easy client 
 interaction with a YARN cluster
  - Application Client protocol: For resource scheduling by apps written in an 
 arbitrary language. Will have to think about throughput concerns
  - ContainerManagement Protocol: Again for arbitrary language apps.
 One important thing to note here is that we already have client libraries on 
 all the three protocols that do some some heavy-lifting. One part of the 
 effort is to figure out if they can be made any thinner and/or how 
 web-services will implement the same functionality.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (YARN-1538) Support web-services for application-submission and AM protocols


 [ 
https://issues.apache.org/jira/browse/YARN-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-1538.
---

Resolution: Duplicate

Closing this as a duplicate of a newer ticket with already created sub-tasks.

 Support web-services for application-submission and AM protocols
 

 Key: YARN-1538
 URL: https://issues.apache.org/jira/browse/YARN-1538
 Project: Hadoop YARN
  Issue Type: New Feature
Reporter: Vinod Kumar Vavilapalli

 We already have read-only web-services for YARN - MAPREDUCE-2863. It'll be 
 great to support APIs for application submission and the AM protocols - 
 scheduling and container-management - as alternatives to RPCs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1714) Per user view in YARN RM

Siqi Li created YARN-1714:
-

 Summary: Per user view in YARN RM
 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Priority: Critical


ResourceManager exposes either one or all jobs via WebUI. It would be good to 
have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1714) Per user view in YARN RM


 [ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1714:
--

Attachment: YARN-1714.v1.patch

 Per user view in YARN RM
 

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1715) Per queue view in RM is not implemented correctly

Siqi Li created YARN-1715:
-

 Summary: Per queue view in RM is not implemented correctly
 Key: YARN-1715
 URL: https://issues.apache.org/jira/browse/YARN-1715
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Siqi Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (YARN-1715) Per queue view in RM is not implemented correctly


 [ 
https://issues.apache.org/jira/browse/YARN-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li reassigned YARN-1715:
-

Assignee: Siqi Li

 Per queue view in RM is not implemented correctly
 -

 Key: YARN-1715
 URL: https://issues.apache.org/jira/browse/YARN-1715
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li





--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1715) Per queue view in RM is not implemented correctly


 [ 
https://issues.apache.org/jira/browse/YARN-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1715:
--

Description: 
For now, per queue view in YARN RM has not yet implemented.
in RmController.java it only set page title for per queue page

 Per queue view in RM is not implemented correctly
 -

 Key: YARN-1715
 URL: https://issues.apache.org/jira/browse/YARN-1715
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li

 For now, per queue view in YARN RM has not yet implemented.
 in RmController.java it only set page title for per queue page



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services

2014-02-11 Thread Alejandro Abdelnur (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898112#comment-13898112
]

Alejandro Abdelnur commented on YARN-1695:
--

Before we start coding work on this, it would be great to see how security will
handled (authentication, ACLs, tokens, etc).

I'm a bit a concern about introducing a second protocol everywhere. From a
maintenance and security risk point of view is doubling our development/support
efforts. Granted, HDFS offers data over RPC and HTTP. But, HTTP, when using
HttpFS (how I recommend using it HTTP access) is a gateway that ends up doing
RPC to HDFS. Thus the only protocol accessing HDFS is RPC.

Have we considered having a C implementation of Hadoop RPC, with the
multiplatform support of protobuffers that may give us the multi-platform
support we are aiming to with a single protocol interface.

Thoughts?

Implement the rest (writable APIs) of RM web-services
-

Key: YARN-1695
URL: https://issues.apache.org/jira/browse/YARN-1695
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli

MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs
added there were only focused on obtaining information from the cluster. We
need to have the following REST APIs to finish the feature
- Application submission/termination (Priority): This unblocks easy client
interaction with a YARN cluster
- Application Client protocol: For resource scheduling by apps written in an
arbitrary language. Will have to think about throughput concerns
- ContainerManagement Protocol: Again for arbitrary language apps.
One important thing to note here is that we already have client libraries on
all the three protocols that do some some heavy-lifting. One part of the
effort is to figure out if they can be made any thinner and/or how
web-services will implement the same functionality.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1714) Per user view in YARN RM


[ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898126#comment-13898126
 ] 

Hadoop QA commented on YARN-1714:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628275/YARN-1714.v1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3066//console

This message is automatically generated.

 Per user view in YARN RM
 

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1714) Per user view in YARN RM


 [ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1714:
--

Attachment: (was: YARN-1714.v1.patch)

 Per user view in YARN RM
 

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1714) Per user view in YARN RM


 [ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1714:
--

Attachment: YARN-1714.v1.patch

 Per user view in YARN RM
 

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1714) Per user view in YARN RM


[ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898146#comment-13898146
 ] 

Hadoop QA commented on YARN-1714:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628283/YARN-1714.v1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3067//console

This message is automatically generated.

 Per user view in YARN RM
 

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1714) Per user view in YARN RM


 [ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1714:
--

Attachment: YARN-1714.v2.patch

 Per user view in YARN RM
 

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1202) Verify RM HA works in secure clusters


 [ 
https://issues.apache.org/jira/browse/YARN-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-1202:
---

Assignee: Wing Yew Poon  (was: Karthik Kambatla)

 Verify RM HA works in secure clusters
 -

 Key: YARN-1202
 URL: https://issues.apache.org/jira/browse/YARN-1202
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Wing Yew Poon

 We should verify RM HA works in secure clusters, and fix any issues that we 
 come across.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1202) Verify RM HA works in secure clusters


[ 
https://issues.apache.org/jira/browse/YARN-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898176#comment-13898176
 ] 

Karthik Kambatla commented on YARN-1202:


[~wypoon] has been doing a lot of RM HA testing on both secure and non-secure 
clusters. Assigning this to him.

Wing Yew - can you report what you found so far? 

 Verify RM HA works in secure clusters
 -

 Key: YARN-1202
 URL: https://issues.apache.org/jira/browse/YARN-1202
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Wing Yew Poon

 We should verify RM HA works in secure clusters, and fix any issues that we 
 come across.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1687) Refactoring timeline classes to remove app related words


 [ 
https://issues.apache.org/jira/browse/YARN-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-1687:
--

Attachment: YARN-1687.1.patch

Created a patch that refactors the timeline module: class names, package names, 
javadoc, variable names, configuration names, webservices paths.

 Refactoring timeline classes to remove app related words
 --

 Key: YARN-1687
 URL: https://issues.apache.org/jira/browse/YARN-1687
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-1687.1.patch


 Remove ATS prefix, change package name, fix javadoc and so on



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1710) Admission Control: agents to allocate reservation


 [ 
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carlo Curino updated YARN-1710:
---

Issue Type: Sub-task  (was: Improvement)
Parent: YARN-1051

 Admission Control: agents to allocate reservation
 -

 Key: YARN-1710
 URL: https://issues.apache.org/jira/browse/YARN-1710
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Carlo Curino

 This JIRA tracks the algorithms used to allocate a user ReservationRequest 
 coming in from the new reservation API (YARN-1708), in the inventory 
 subsystem (YARN-1709) maintaining the current plan for the cluster. The focus 
 of this agents is to quickly find a solution for the set of contraints 
 provided by the user, and the physical constraints of the plan.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1707) Making the CapacityScheduler more dynamic


 [ 
https://issues.apache.org/jira/browse/YARN-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carlo Curino updated YARN-1707:
---

Issue Type: Sub-task  (was: Improvement)
Parent: YARN-1051

 Making the CapacityScheduler more dynamic
 -

 Key: YARN-1707
 URL: https://issues.apache.org/jira/browse/YARN-1707
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: capacityscheduler
Reporter: Carlo Curino
Assignee: Carlo Curino

 The CapacityScheduler is a rather static at the moment, and refreshqueue 
 provides a rather heavy-handed way to reconfigure it. Moving towards 
 long-running services (tracked in YARN-896) and to enable more advanced 
 admission control and resource parcelling we need to make the 
 CapacityScheduler more dynamic. This is instrumental to the umbrella jira 
 YARN-1051.
 Concretely this require the following changes:
 * create queues dynamically
 * destroy queues dynamically
 * dynamically change queue parameters (e.g., capacity) 
 * modify refreshqueue validation to enforce sum(child.getCapacity())= 100% 
 instead of ==100%
 We limit this to LeafQueues. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1708) Add a public API to reserve resources (part of YARN-1051)


 [ 
https://issues.apache.org/jira/browse/YARN-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carlo Curino updated YARN-1708:
---

Issue Type: Sub-task  (was: Improvement)
Parent: YARN-1051

 Add a public API to reserve resources (part of YARN-1051)
 -

 Key: YARN-1708
 URL: https://issues.apache.org/jira/browse/YARN-1708
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Carlo Curino

 This JIRA tracks the definition of a new public API for YARN, which allows 
 users to reserve resources (think of time-bounded queues). This is part of 
 the admission control enhancement proposed in YARN-1051.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1709) Admission Control: inventory subsystem


 [ 
https://issues.apache.org/jira/browse/YARN-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carlo Curino updated YARN-1709:
---

Issue Type: Sub-task  (was: Improvement)
Parent: YARN-1051

 Admission Control: inventory subsystem
 --

 Key: YARN-1709
 URL: https://issues.apache.org/jira/browse/YARN-1709
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Carlo Curino

 This JIRA is about the key data structure used to track resources over time 
 to enable YARN-1051. The inventory subsystem is conceptually a plan of how 
 the capacity scheduler will be configured over-time.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709


 [ 
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carlo Curino updated YARN-1711:
---

Issue Type: Sub-task  (was: Improvement)
Parent: YARN-1051

 CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
 --

 Key: YARN-1711
 URL: https://issues.apache.org/jira/browse/YARN-1711
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino

 This JIRA tracks the development of a policy that enforces user quotas (a 
 time-extension of the notion of capacity) in the inventory subsystem 
 discussed in YARN-1709.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1712) Admission Control: plan follower


 [ 
https://issues.apache.org/jira/browse/YARN-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carlo Curino updated YARN-1712:
---

Issue Type: Sub-task  (was: Improvement)
Parent: YARN-1051

 Admission Control: plan follower
 

 Key: YARN-1712
 URL: https://issues.apache.org/jira/browse/YARN-1712
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: capacityscheduler, resourcemanager
Reporter: Carlo Curino
Assignee: Carlo Curino

 This JIRA tracks a thread that continuously propagates the current state of 
 an inventory subsystem to the scheduler. As the inventory subsystem store the 
 plan of how the resources should be subdivided, the work we propose in this 
 JIRA realizes such plan by dynamically instructing the CapacityScheduler to 
 add/remove/resize queues to follow the plan.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (YARN-1708) Add a public API to reserve resources (part of YARN-1051)


 [ 
https://issues.apache.org/jira/browse/YARN-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carlo Curino reassigned YARN-1708:
--

Assignee: Subramaniam Krishnan

 Add a public API to reserve resources (part of YARN-1051)
 -

 Key: YARN-1708
 URL: https://issues.apache.org/jira/browse/YARN-1708
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Carlo Curino
Assignee: Subramaniam Krishnan

 This JIRA tracks the definition of a new public API for YARN, which allows 
 users to reserve resources (think of time-bounded queues). This is part of 
 the admission control enhancement proposed in YARN-1051.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (YARN-1709) Admission Control: inventory subsystem

2014-02-11 Thread Subramaniam Krishnan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subramaniam Krishnan reassigned YARN-1709:
--

Assignee: Subramaniam Krishnan

 Admission Control: inventory subsystem
 --

 Key: YARN-1709
 URL: https://issues.apache.org/jira/browse/YARN-1709
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Carlo Curino
Assignee: Subramaniam Krishnan

 This JIRA is about the key data structure used to track resources over time 
 to enable YARN-1051. The inventory subsystem is conceptually a plan of how 
 the capacity scheduler will be configured over-time.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (YARN-1710) Admission Control: agents to allocate reservation

2014-02-11 Thread Subramaniam Krishnan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subramaniam Krishnan reassigned YARN-1710:
--

Assignee: Subramaniam Krishnan

 Admission Control: agents to allocate reservation
 -

 Key: YARN-1710
 URL: https://issues.apache.org/jira/browse/YARN-1710
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Carlo Curino
Assignee: Subramaniam Krishnan

 This JIRA tracks the algorithms used to allocate a user ReservationRequest 
 coming in from the new reservation API (YARN-1708), in the inventory 
 subsystem (YARN-1709) maintaining the current plan for the cluster. The focus 
 of this agents is to quickly find a solution for the set of contraints 
 provided by the user, and the physical constraints of the plan.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.

2014-02-11 Thread Subramaniam Krishnan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subramaniam Krishnan updated YARN-1051:
---

Attachment: YARN-1051-design.pdf

Attaching the approach doc that describes the overall intent for interested 
readers. The doc also lists the breakdown into incremental sub-tasks.Any 
suggestions/thoughts are welcome, we will incorporate feedback as it comes in.  

 YARN Admission Control/Planner: enhancing the resource allocation model with 
 time.
 --

 Key: YARN-1051
 URL: https://issues.apache.org/jira/browse/YARN-1051
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: capacityscheduler, resourcemanager, scheduler
Reporter: Carlo Curino
Assignee: Carlo Curino
 Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf


 In this umbrella JIRA we propose to extend the YARN RM to handle time 
 explicitly, allowing users to reserve capacity over time. This is an 
 important step towards SLAs, long-running services, workflows, and helps for 
 gang scheduling.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1697) NodeManager reports negative running containers


[ 
https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898309#comment-13898309
 ] 

Hudson commented on YARN-1697:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5148 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5148/])
YARN-1697. NodeManager reports negative running containers (Sandy Ryza) (sandy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567356)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java


 NodeManager reports negative running containers
 ---

 Key: YARN-1697
 URL: https://issues.apache.org/jira/browse/YARN-1697
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.3.0
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.4.0

 Attachments: YARN-1697.patch


 We're seeing the NodeManager metrics report a negative number of running 
 containers.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1714) Per user view in YARN RM


[ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898314#comment-13898314
 ] 

Hadoop QA commented on YARN-1714:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628295/YARN-1714.v2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3068//console

This message is automatically generated.

 Per user view in YARN RM
 

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1690) sending ATS events from Distributed shell


[ 
https://issues.apache.org/jira/browse/YARN-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898329#comment-13898329
 ] 

Zhijie Shen commented on YARN-1690:
---

TimelineClient is ready. The patch is stale now.

 sending ATS events from Distributed shell 
 --

 Key: YARN-1690
 URL: https://issues.apache.org/jira/browse/YARN-1690
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Mayank Bansal
 Attachments: YARN-1690-1.patch, YARN-1690-2.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1676) Make admin refreshUserToGroupsMappings of configuration work across RM failover

2014-02-11 Thread Xuan Gong (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-1676:


Attachment: YARN-1676.4.patch

Modify the testcase to call updateGroups() to update groups before calling 
refreshUserToGroupsMappings. After that, verify whether the groups are updated.

 Make admin refreshUserToGroupsMappings of configuration work across RM 
 failover
 ---

 Key: YARN-1676
 URL: https://issues.apache.org/jira/browse/YARN-1676
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
 Attachments: YARN-1667.3.patch, YARN-1676.1.patch, YARN-1676.2.patch, 
 YARN-1676.3.patch, YARN-1676.4.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (YARN-1682) TestRMRestart#testRMRestartSucceededApp occasionally fails


 [ 
https://issues.apache.org/jira/browse/YARN-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved YARN-1682.
--

Resolution: Cannot Reproduce

 TestRMRestart#testRMRestartSucceededApp occasionally fails
 --

 Key: YARN-1682
 URL: https://issues.apache.org/jira/browse/YARN-1682
 Project: Hadoop YARN
  Issue Type: Test
Reporter: Ted Yu

 From https://builds.apache.org/job/Hadoop-Yarn-trunk/468/console :
 {code}
 testRMRestartSucceededApp(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
   Time elapsed: 8.129 sec   FAILURE!
 java.lang.AssertionError: null
   at org.junit.Assert.fail(Assert.java:92)
   at org.junit.Assert.assertTrue(Assert.java:43)
   at org.junit.Assert.assertTrue(Assert.java:54)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.verifyAppReportAfterRMRestart(TestRMRestart.java:900)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartSucceededApp(TestRMRestart.java:774)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1515) Ability to dump the container threads and stop the containers in a single RPC


[ 
https://issues.apache.org/jira/browse/YARN-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898425#comment-13898425
 ] 

Karthik Kambatla commented on YARN-1515:


I just barely skimmed through the patch. Before taking a closer look at it, can 
we consolidate the new ContainerManagementProtocol#stopContainers1 and the 
existing ContainerManagementProtocol#stopContainers? That would probably make 
the patch shorter too - easier to review :)

 Ability to dump the container threads and stop the containers in a single RPC
 -

 Key: YARN-1515
 URL: https://issues.apache.org/jira/browse/YARN-1515
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api, nodemanager
Reporter: Gera Shegalov
Assignee: Gera Shegalov
 Attachments: YARN-1515.v01.patch, YARN-1515.v02.patch


 This is needed to implement MAPREDUCE-5044 to enable thread diagnostics for 
 timed-out task attempts.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper


[ 
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898433#comment-13898433
 ] 

Ted Yu commented on YARN-1296:
--

Is more review needed ?

 schedulerAllocateTimer is accessed without holding samplerLock in 
 ResourceSchedulerWrapper
 --

 Key: YARN-1296
 URL: https://issues.apache.org/jira/browse/YARN-1296
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Attachments: yarn-1296-v1.patch


 Here is related code:
 {code}
   public Allocation allocate(ApplicationAttemptId attemptId,
  ListResourceRequest resourceRequests,
  ListContainerId containerIds,
  ListString strings, ListString strings2) {
 if (metricsON) {
   final Timer.Context context = schedulerAllocateTimer.time();
 {code}
 samplerLock should be used to guard the access.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1583) Ineffective state check in FairSchedulerAppsBlock#render()


 [ 
https://issues.apache.org/jira/browse/YARN-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated YARN-1583:
-

Priority: Major  (was: Minor)

 Ineffective state check in FairSchedulerAppsBlock#render()
 --

 Key: YARN-1583
 URL: https://issues.apache.org/jira/browse/YARN-1583
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Ted Yu

 Starting line 90:
 {code}
 for (RMApp app : apps.values()) {
   if (reqAppStates != null  !reqAppStates.contains(app.getState())) {
 {code}
 reqAppStates is of type YarnApplicationState.
 app.getState() returns RMAppState.
 These are two different enum types.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Moved] (YARN-1716) finishedStatus is accessed without proper locking in RMContainerImpl#FinishedTransition#transition()


 [ 
https://issues.apache.org/jira/browse/YARN-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu moved MAPREDUCE-5753 to YARN-1716:
-

Key: YARN-1716  (was: MAPREDUCE-5753)
Project: Hadoop YARN  (was: Hadoop Map/Reduce)

 finishedStatus is accessed without proper locking in 
 RMContainerImpl#FinishedTransition#transition()
 

 Key: YARN-1716
 URL: https://issues.apache.org/jira/browse/YARN-1716
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Ted Yu

 {code}
   container.finishedStatus = finishedEvent.getRemoteContainerStatus();
 {code}
 The access should be guarded by obtaining readLock



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1714) Per user and per queue view in YARN RM


 [ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1714:
--

Summary: Per user and per queue view in YARN RM  (was: Per user view in 
YARN RM)

 Per user and per queue view in YARN RM
 --

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1525) Web UI should redirect to active RM when HA is enabled.

2014-02-11 Thread Cindy Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cindy Li updated YARN-1525:
---

Attachment: YARN1525.v9.patch

Xuan and Karthik, I've included changes according to your comments: added 
security enabling check, minimized format related changes, changed comments for 
test case.  Thanks.

 Web UI should redirect to active RM when HA is enabled.
 ---

 Key: YARN-1525
 URL: https://issues.apache.org/jira/browse/YARN-1525
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
 Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, 
 YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch


 When failover happens, web UI should redirect to the current active rm.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1714) Per user and per queue view in YARN RM


 [ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1714:
--

Description: 
ResourceManager exposes either one or all jobs via WebUI. It would be good to 
have filter for user so that see only their jobs.

Provide rest style url to access only user specified queue or user apps. 
For instance,
http://hadoop-example.com:50030/cluster/user/toto   
  displays apps owned by toto
http://hadoop-example.com:50030/cluster/user/toto,glinda
  displays apps owned by toto and glinda
http://hadoop-example.com:50030/cluster/queue/root.queue1   
 displays apps in root.queue1
http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2   
displays apps in root.queue1 and  root.queue2

  was:ResourceManager exposes either one or all jobs via WebUI. It would be 
good to have filter for user so that see only their jobs.


 Per user and per queue view in YARN RM
 --

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.
 Provide rest style url to access only user specified queue or user apps. 
 For instance,
 http://hadoop-example.com:50030/cluster/user/toto 
 displays apps owned by toto
 http://hadoop-example.com:50030/cluster/user/toto,glinda  
 displays apps owned by toto and glinda
 http://hadoop-example.com:50030/cluster/queue/root.queue1 
displays apps in root.queue1
 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2   
 displays apps in root.queue1 and  root.queue2



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper


[ 
https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898475#comment-13898475
 ] 

Hadoop QA commented on YARN-1296:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12607939/yarn-1296-v1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-sls.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3070//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3070//console

This message is automatically generated.

 schedulerAllocateTimer is accessed without holding samplerLock in 
 ResourceSchedulerWrapper
 --

 Key: YARN-1296
 URL: https://issues.apache.org/jira/browse/YARN-1296
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Attachments: yarn-1296-v1.patch


 Here is related code:
 {code}
   public Allocation allocate(ApplicationAttemptId attemptId,
  ListResourceRequest resourceRequests,
  ListContainerId containerIds,
  ListString strings, ListString strings2) {
 if (metricsON) {
   final Timer.Context context = schedulerAllocateTimer.time();
 {code}
 samplerLock should be used to guard the access.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.


[ 
https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898478#comment-13898478
 ] 

Karthik Kambatla commented on YARN-1051:


Thanks Carlo and Subru for sharing the TR and design-doc.

Can you please verify my understanding is right (haven't read the TR yet, just 
the design doc): The {{Admission Control}} box (Planning agent and Inventory) 
takes care of figuring out if a reservation is feasible and managing the 
corresponding inventory. Plan Follower would take care of actually submitting 
the applications to the RM (scheduler) and scheduler-queue configurations come 
show time; the updated scheduler-queue configurations would ensure these apps 
would actually get the resources they need. 

 YARN Admission Control/Planner: enhancing the resource allocation model with 
 time.
 --

 Key: YARN-1051
 URL: https://issues.apache.org/jira/browse/YARN-1051
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: capacityscheduler, resourcemanager, scheduler
Reporter: Carlo Curino
Assignee: Carlo Curino
 Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf


 In this umbrella JIRA we propose to extend the YARN RM to handle time 
 explicitly, allowing users to reserve capacity over time. This is an 
 important step towards SLAs, long-running services, workflows, and helps for 
 gang scheduling.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.


[ 
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898487#comment-13898487
 ] 

Karthik Kambatla commented on YARN-1525:


Nits before actual review:
# webapp/Dispatcher.java and webapp/Dispatcher.java have spurious changes to 
the license header - may be IDE induced. 
# The methods in RMHAUtils can be moved to HAUtil. It would be nice to get this 
done soon, so these are also taken care of by Vinod's attempt to clean up 
HAUtil. Sorry for the additional trouble, Vinod. 



 Web UI should redirect to active RM when HA is enabled.
 ---

 Key: YARN-1525
 URL: https://issues.apache.org/jira/browse/YARN-1525
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
 Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, 
 YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch


 When failover happens, web UI should redirect to the current active rm.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt


 [ 
https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-1588:
--

Attachment: YARN-1588.1.patch

 Rebind NM tokens for previous attempt's running containers to the new attempt
 -

 Key: YARN-1588
 URL: https://issues.apache.org/jira/browse/YARN-1588
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
 Attachments: YARN-1588.1.patch, YARN-1588.1.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.


[ 
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898488#comment-13898488
 ] 

Karthik Kambatla commented on YARN-1525:


Oh - missed these - would be nice to not include these either:
{code}
-@InterfaceAudience.LimitedPrivate({YARN, MapReduce})
+@InterfaceAudience.LimitedPrivate({ YARN, MapReduce })
 public abstract class WebApp extends ServletModule {
   private static final Logger LOG = LoggerFactory.getLogger(WebApp.class);
 
-  public enum HTTP { GET, POST, HEAD, PUT, DELETE };
+  public enum HTTP {
+GET, POST, HEAD, PUT, DELETE
+  };
{code}

 Web UI should redirect to active RM when HA is enabled.
 ---

 Key: YARN-1525
 URL: https://issues.apache.org/jira/browse/YARN-1525
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
 Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, 
 YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch


 When failover happens, web UI should redirect to the current active rm.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt


 [ 
https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He reassigned YARN-1588:
-

Assignee: Jian He

 Rebind NM tokens for previous attempt's running containers to the new attempt
 -

 Key: YARN-1588
 URL: https://issues.apache.org/jira/browse/YARN-1588
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-1588.1.patch, YARN-1588.1.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1676) Make admin refreshUserToGroupsMappings of configuration work across RM failover


[ 
https://issues.apache.org/jira/browse/YARN-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898496#comment-13898496
 ] 

Hadoop QA commented on YARN-1676:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628338/YARN-1676.4.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3069//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3069//console

This message is automatically generated.

 Make admin refreshUserToGroupsMappings of configuration work across RM 
 failover
 ---

 Key: YARN-1676
 URL: https://issues.apache.org/jira/browse/YARN-1676
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
 Attachments: YARN-1667.3.patch, YARN-1676.1.patch, YARN-1676.2.patch, 
 YARN-1676.3.patch, YARN-1676.4.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.


[ 
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898497#comment-13898497
 ] 

Hadoop QA commented on YARN-1525:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628362/YARN1525.v9.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3071//console

This message is automatically generated.

 Web UI should redirect to active RM when HA is enabled.
 ---

 Key: YARN-1525
 URL: https://issues.apache.org/jira/browse/YARN-1525
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
 Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, 
 YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch


 When failover happens, web UI should redirect to the current active rm.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1714) Per user and per queue view in YARN RM


 [ 
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1714:
--

Attachment: YARN-1714.v3.patch

 Per user and per queue view in YARN RM
 --

 Key: YARN-1714
 URL: https://issues.apache.org/jira/browse/YARN-1714
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
 Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, 
 YARN-1714.v3.patch


 ResourceManager exposes either one or all jobs via WebUI. It would be good to 
 have filter for user so that see only their jobs.
 Provide rest style url to access only user specified queue or user apps. 
 For instance,
 http://hadoop-example.com:50030/cluster/user/toto 
 displays apps owned by toto
 http://hadoop-example.com:50030/cluster/user/toto,glinda  
 displays apps owned by toto and glinda
 http://hadoop-example.com:50030/cluster/queue/root.queue1 
displays apps in root.queue1
 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2   
 displays apps in root.queue1 and  root.queue2



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.

2014-02-11 Thread Cindy Li (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898500#comment-13898500
 ] 

Cindy Li commented on YARN-1525:


@Karthik, RMHAUtils.java couldn't be merged into HAUtil.java because of project 
setup, as it uses RMHAServiceTarget which would not be accessible from 
HAUtil.java. Yes. the format are automatically changed by IDE. I can try to 
manually remove them.

 Web UI should redirect to active RM when HA is enabled.
 ---

 Key: YARN-1525
 URL: https://issues.apache.org/jira/browse/YARN-1525
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Cindy Li
 Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, 
 YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch


 When failover happens, web UI should redirect to the current active rm.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1717) Misc improvements to leveldb timeline store

Billie Rinaldi created YARN-1717:


 Summary: Misc improvements to leveldb timeline store
 Key: YARN-1717
 URL: https://issues.apache.org/jira/browse/YARN-1717
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi


The leveldb timeline store implementation needs the following:
* better documentation of its internal structures
* braces for all control flow statements
* simple locking to prevent issues related to concurrent writes
* internal changes to enable deleting entities
** never overwrite existing primary filter entries
** add hidden reverse pointers to related entities



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1492) truly shared cache for jars (jobjar/libjar)

2014-02-11 Thread Chris Trezzo (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Trezzo updated YARN-1492:
---

Attachment: shared_cache_design_v5.pdf

Attached is a v5 of the design document. Some of the major changes are:
# Cache management logic moved from the client to a shared cache manager and
localization service. As a benefit, cached resources are now read-only and
managed solely by a trusted source (SCM and NodeManager) running as its own
user.
# Caching of a resource is now off the job submission path and happens
asynchronously. If submitting to the cache fails, it does not affect the job.
# Improved robustness when dealing with clients that don’t clean up after
themselves correctly.

Feedback is always appreciated. I will be starting implementation shortly.

truly shared cache for jars (jobjar/libjar)
---

Key: YARN-1492
URL: https://issues.apache.org/jira/browse/YARN-1492
Project: Hadoop YARN
Issue Type: New Feature
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: shared_cache_design.pdf, shared_cache_design_v2.pdf,
shared_cache_design_v3.pdf, shared_cache_design_v4.pdf,
shared_cache_design_v5.pdf

Currently there is the distributed cache that enables you to cache jars and
files so that attempts from the same job can reuse them. However, sharing is
limited with the distributed cache because it is normally on a per-job basis.
On a large cluster, sometimes copying of jobjars and libjars becomes so
prevalent that it consumes a large portion of the network bandwidth, not to
speak of defeating the purpose of bringing compute to where data is. This
is wasteful because in most cases code doesn't change much across many jobs.
I'd like to propose and discuss feasibility of introducing a truly shared
cache so that multiple jobs from multiple users can share and cache jars.
This JIRA is to open the discussion.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM

[
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898512#comment-13898512
]

Hadoop QA commented on YARN-1714:
-

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628368/YARN-1714.v3.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3073//console

This message is automatically generated.

Per user and per queue view in YARN RM
--

Key: YARN-1714
URL: https://issues.apache.org/jira/browse/YARN-1714
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch,
YARN-1714.v3.patch

ResourceManager exposes either one or all jobs via WebUI. It would be good to
have filter for user so that see only their jobs.
Provide rest style url to access only user specified queue or user apps.
For instance,
http://hadoop-example.com:50030/cluster/user/toto
displays apps owned by toto
http://hadoop-example.com:50030/cluster/user/toto,glinda
displays apps owned by toto and glinda
http://hadoop-example.com:50030/cluster/queue/root.queue1
displays apps in root.queue1
http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2
displays apps in root.queue1 and root.queue2

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1717) Misc improvements to leveldb timeline store


 [ 
https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-1717:
-

Attachment: YARN-1717.1.patch

Here is my first attempt at this patch.  If desired, I can remove the new 
methods appearing at the end of the leveldb impl that enable offline entity age 
off and put those on a separate ticket (i.e. getEntityTypes, removeStartTime, 
deleteKeysWithPrefix, deleteEntity, deleteOldEntries).

 Misc improvements to leveldb timeline store
 ---

 Key: YARN-1717
 URL: https://issues.apache.org/jira/browse/YARN-1717
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1717.1.patch


 The leveldb timeline store implementation needs the following:
 * better documentation of its internal structures
 * braces for all control flow statements
 * simple locking to prevent issues related to concurrent writes
 * internal changes to enable deleting entities
 ** never overwrite existing primary filter entries
 ** add hidden reverse pointers to related entities



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt


[ 
https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898537#comment-13898537
 ] 

Hadoop QA commented on YARN-1588:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628365/YARN-1588.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 3 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3072//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/3072//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3072//console

This message is automatically generated.

 Rebind NM tokens for previous attempt's running containers to the new attempt
 -

 Key: YARN-1588
 URL: https://issues.apache.org/jira/browse/YARN-1588
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-1588.1.patch, YARN-1588.1.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules

Sandy Ryza created YARN-1718:


 Summary: Fix a couple isTerminals in Fair Scheduler queue 
placement rules 
 Key: YARN-1718
 URL: https://issues.apache.org/jira/browse/YARN-1718
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Reporter: Sandy Ryza
Assignee: Sandy Ryza


SecondaryGroupExistingQueue and Default are incorrect



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules


 [ 
https://issues.apache.org/jira/browse/YARN-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated YARN-1718:
-

Attachment: YARN-1718.patch

 Fix a couple isTerminals in Fair Scheduler queue placement rules 
 -

 Key: YARN-1718
 URL: https://issues.apache.org/jira/browse/YARN-1718
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-1718.patch


 SecondaryGroupExistingQueue and Default are incorrect



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt


 [ 
https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-1588:
--

Attachment: YARN-1588.2.patch

 Rebind NM tokens for previous attempt's running containers to the new attempt
 -

 Key: YARN-1588
 URL: https://issues.apache.org/jira/browse/YARN-1588
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1719) ATSWebServices produces jersey warnings

Billie Rinaldi created YARN-1719:


 Summary: ATSWebServices produces jersey warnings
 Key: YARN-1719
 URL: https://issues.apache.org/jira/browse/YARN-1719
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1719.1.patch

These don't appear to affect how the web services work, but the following 
warnings are logged:
{noformat}
WARNING: The following warnings have been detected with resource and/or provider
 classes:
  WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati
onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a
pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, is t
reated as a resource method
  WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt
imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web
app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
ntities), with URI template, /, is treated as a resource method
{noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1719) ATSWebServices produces jersey warnings


 [ 
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-1719:
-

Attachment: YARN-1719.1.patch

 ATSWebServices produces jersey warnings
 ---

 Key: YARN-1719
 URL: https://issues.apache.org/jira/browse/YARN-1719
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1719.1.patch


 These don't appear to affect how the web services work, but the following 
 warnings are logged:
 {noformat}
 WARNING: The following warnings have been detected with resource and/or 
 provider
  classes:
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.server.applicati
 onhistoryservice.webapp.ATSWebServices$AboutInfo 
 org.apache.hadoop.yarn.server.a
 pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
 letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, 
 is t
 reated as a resource method
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.api.records.appt
 imeline.ATSPutErrors 
 org.apache.hadoop.yarn.server.applicationhistoryservice.web
 app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
 let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
 ntities), with URI template, /, is treated as a resource method
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1495) Allow moving apps between queues

[
https://issues.apache.org/jira/browse/YARN-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898568#comment-13898568
]

Sandy Ryza commented on YARN-1495:
--

bq. From an API point of view, there should be a way for the application at
run-time/registration time to find out what features are supported or not
supported by the currently configured scheduler in the RM.
That makes total sense to me.

bq. Now, for moving apps across schedulers - given that it is a client only
feature and there is no changes required in an application, my previous
comment's argument does not hold for this feature.
Great. Will go forward with the merge to branch-2.

Allow moving apps between queues

Key: YARN-1495
URL: https://issues.apache.org/jira/browse/YARN-1495
Project: Hadoop YARN
Issue Type: New Feature
Components: scheduler
Affects Versions: 2.2.0
Reporter: Sandy Ryza
Assignee: Sandy Ryza

This is an umbrella JIRA for work needed to allow moving YARN applications
from one queue to another. The work will consist of additions in the command
line options, additions in the client RM protocol, and changes in the
schedulers to support this.
I have a picture of how this should function in the Fair Scheduler, but I'm
not familiar enough with the Capacity Scheduler for the same there.
Ultimately, the decision to whether an application can be moved should go
down to the scheduler - some schedulers may wish not to support this at all.
However, schedulers that do support it should share some common semantics
around ACLs and what happens to running containers.
Here is how I see the general semantics working out:
* A move request is issued by the client. After it gets past ACLs, the
scheduler checks whether executing the move will violate any constraints. For
the Fair Scheduler, these would be queue maxRunningApps and queue
maxResources constraints
* All running containers are transferred from the old queue to the new queue
* All outstanding requests are transferred from the old queue to the new queue
Here is I see the ACLs of this working out:
* To move an app from a queue a user must have modify access on the app or
administer access on the queue
* To move an app to a queue a user must have submit access on the queue or
administer access on the queue

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1676) Make admin refreshUserToGroupsMappings of configuration work across RM failover


[ 
https://issues.apache.org/jira/browse/YARN-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898571#comment-13898571
 ] 

Vinod Kumar Vavilapalli commented on YARN-1676:
---

+1, this looks good. Checking this in.

 Make admin refreshUserToGroupsMappings of configuration work across RM 
 failover
 ---

 Key: YARN-1676
 URL: https://issues.apache.org/jira/browse/YARN-1676
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
 Attachments: YARN-1667.3.patch, YARN-1676.1.patch, YARN-1676.2.patch, 
 YARN-1676.3.patch, YARN-1676.4.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1720) QueuePlacementRule.SecondaryGroupExistingQueue should not be terminal

2014-02-11 Thread Aditya Acharya (JIRA)

Aditya Acharya created YARN-1720:


 Summary: QueuePlacementRule.SecondaryGroupExistingQueue should not 
be terminal
 Key: YARN-1720
 URL: https://issues.apache.org/jira/browse/YARN-1720
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.2.0
Reporter: Aditya Acharya
Assignee: Sandy Ryza


The SecondaryGroupExistingQueue QueuePlacementRule's isTerminal() method should 
always return false, not create, because it does not, in fact, create a queue 
ever.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN

2014-02-11 Thread Haohui Mai (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated YARN-1553:
-

Attachment: YARN-1553.006.patch

The v6 patch ensures that the AM always starts an HTTP server. Thanks 
[~vinodkv] for hunting the bug.

 Do not use HttpConfig.isSecure() in YARN
 

 Key: YARN-1553
 URL: https://issues.apache.org/jira/browse/YARN-1553
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: YARN-1553.000.patch, YARN-1553.001.patch, 
 YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch, 
 YARN-1553.005.patch, YARN-1553.006.patch


 HDFS-5305 and related jira decide that each individual project will have 
 their own configuration on http policy. {{HttpConfig.isSecure}} is a global 
 static method which does not fit the design anymore. The same functionality 
 should be moved into the YARN code base.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1717) Misc improvements to leveldb timeline store


[ 
https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898575#comment-13898575
 ] 

Hadoop QA commented on YARN-1717:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628373/YARN-1717.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3074//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3074//console

This message is automatically generated.

 Misc improvements to leveldb timeline store
 ---

 Key: YARN-1717
 URL: https://issues.apache.org/jira/browse/YARN-1717
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1717.1.patch


 The leveldb timeline store implementation needs the following:
 * better documentation of its internal structures
 * braces for all control flow statements
 * simple locking to prevent issues related to concurrent writes
 * internal changes to enable deleting entities
 ** never overwrite existing primary filter entries
 ** add hidden reverse pointers to related entities



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings


[ 
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898599#comment-13898599
 ] 

Hadoop QA commented on YARN-1719:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628389/YARN-1719.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3076//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3076//console

This message is automatically generated.

 ATSWebServices produces jersey warnings
 ---

 Key: YARN-1719
 URL: https://issues.apache.org/jira/browse/YARN-1719
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1719.1.patch


 These don't appear to affect how the web services work, but the following 
 warnings are logged:
 {noformat}
 WARNING: The following warnings have been detected with resource and/or 
 provider
  classes:
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.server.applicati
 onhistoryservice.webapp.ATSWebServices$AboutInfo 
 org.apache.hadoop.yarn.server.a
 pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
 letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, 
 is t
 reated as a resource method
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.api.records.appt
 imeline.ATSPutErrors 
 org.apache.hadoop.yarn.server.applicationhistoryservice.web
 app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
 let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
 ntities), with URI template, /, is treated as a resource method
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1495) Allow moving apps between queues

2014-02-11 Thread Hitesh Shah (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898601#comment-13898601
 ] 

Hitesh Shah commented on YARN-1495:
---

[~sandyr] Just to be clear, I have not reviewed the code as such so please do 
not consider my comment as a +1 if that is required for the merge. I am 
assuming there are others who are more familiar with these changes and have 
reviewed/+1'ed it for the merge. 

 Allow moving apps between queues
 

 Key: YARN-1495
 URL: https://issues.apache.org/jira/browse/YARN-1495
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: scheduler
Affects Versions: 2.2.0
Reporter: Sandy Ryza
Assignee: Sandy Ryza

 This is an umbrella JIRA for work needed to allow moving YARN applications 
 from one queue to another.  The work will consist of additions in the command 
 line options, additions in the client RM protocol, and changes in the 
 schedulers to support this.
 I have a picture of how this should function in the Fair Scheduler, but I'm 
 not familiar enough with the Capacity Scheduler for the same there.  
 Ultimately, the decision to whether an application can be moved should go 
 down to the scheduler - some schedulers may wish not to support this at all.  
 However, schedulers that do support it should share some common semantics 
 around ACLs and what happens to running containers.
 Here is how I see the general semantics working out:
 * A move request is issued by the client.  After it gets past ACLs, the 
 scheduler checks whether executing the move will violate any constraints. For 
 the Fair Scheduler, these would be queue maxRunningApps and queue 
 maxResources constraints
 * All running containers are transferred from the old queue to the new queue
 * All outstanding requests are transferred from the old queue to the new queue
 Here is I see the ACLs of this working out:
 * To move an app from a queue a user must have modify access on the app or 
 administer access on the queue
 * To move an app to a queue a user must have submit access on the queue or 
 administer access on the queue 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.


 [ 
https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He reassigned YARN-1417:
-

Assignee: Jian He  (was: Omkar Vinit Joshi)

 RM may issue expired container tokens to AM while issuing new containers.
 -

 Key: YARN-1417
 URL: https://issues.apache.org/jira/browse/YARN-1417
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Jian He
Priority: Blocker
 Attachments: YARN-1417.2.patch


 Today we create new container token when we create container in RM as a part 
 of schedule cycle. However that container may get reserved or assigned. If 
 the container gets reserved and remains like that (in reserved state) for 
 more than container token expiry interval then RM will end up issuing 
 container with expired token.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules


[ 
https://issues.apache.org/jira/browse/YARN-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898628#comment-13898628
 ] 

Hadoop QA commented on YARN-1718:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628387/YARN-1718.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3075//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3075//console

This message is automatically generated.

 Fix a couple isTerminals in Fair Scheduler queue placement rules 
 -

 Key: YARN-1718
 URL: https://issues.apache.org/jira/browse/YARN-1718
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-1718.patch


 SecondaryGroupExistingQueue and Default are incorrect



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt


[ 
https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898631#comment-13898631
 ] 

Hadoop QA commented on YARN-1588:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628388/YARN-1588.2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3077//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/3077//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3077//console

This message is automatically generated.

 Rebind NM tokens for previous attempt's running containers to the new attempt
 -

 Key: YARN-1588
 URL: https://issues.apache.org/jira/browse/YARN-1588
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1641) ZK store should attempt a write periodically to ensure it is still Active


[ 
https://issues.apache.org/jira/browse/YARN-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898633#comment-13898633
 ] 

Sandy Ryza commented on YARN-1641:
--

The added thread should be given a name.  Otherwise, LGTM.

 ZK store should attempt a write periodically to ensure it is still Active
 -

 Key: YARN-1641
 URL: https://issues.apache.org/jira/browse/YARN-1641
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.3.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: yarn-1641-1.patch


 Fencing in ZK store kicks in when the RM tries to write something to the 
 store. If the RM doesn't write anything to the store, it doesn't get fenced 
 and can continue to assume being the Active. 
 By periodically writing a file (say, every RM_ZK_TIMEOUT_MS seconds), we can 
 ensure it gets fenced.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1717) Misc improvements to leveldb timeline store


 [ 
https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-1717:
-

Attachment: YARN-1717.2.patch

Fixed findbugs warning.

 Misc improvements to leveldb timeline store
 ---

 Key: YARN-1717
 URL: https://issues.apache.org/jira/browse/YARN-1717
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1717.1.patch, YARN-1717.2.patch


 The leveldb timeline store implementation needs the following:
 * better documentation of its internal structures
 * braces for all control flow statements
 * simple locking to prevent issues related to concurrent writes
 * internal changes to enable deleting entities
 ** never overwrite existing primary filter entries
 ** add hidden reverse pointers to related entities



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1495) Allow moving apps between queues

[
https://issues.apache.org/jira/browse/YARN-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898666#comment-13898666
]

Karthik Kambatla commented on YARN-1495:

None of the patches introduce any Public-Stable APIs. I think it is reasonable
to merge them to branch-2. +1. Thanks Sandy.

Allow moving apps between queues

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM

[
https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898669#comment-13898669
]

Zhijie Shen commented on YARN-1714:
---

Hi Siqi, I've some thoughts about this issue:

1. Actually the current apps page provide users the full text search over the
app list. Therefore, you can input user/queue to retrieve the sub list of a
certain user/queue. Though I understand it's not perfect way as a complete list
has to be retrieved anyway, it should meet your basic requirement, shouldn't
it? Or you're looking for some thing efficient?

2. Another consideration is that now REST API is providing more filters
(including user/queue) than web UI to get applications. Ideally, the two
interfaces should have same or equivalent capability of exposing data. Probably
you want to think about other filters here as well.

3. I had a quick look at the patch. I didn't see there's an iterative way for
users to do the query other than editing url. Did I miss something?

4. Should we reuse AppsBlock?

Per user and per queue view in YARN RM
--

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings


[ 
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898673#comment-13898673
 ] 

Zhijie Shen commented on YARN-1719:
---

+1. Verified locally, the warnings were gone. Will commit it.

 ATSWebServices produces jersey warnings
 ---

 Key: YARN-1719
 URL: https://issues.apache.org/jira/browse/YARN-1719
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1719.1.patch


 These don't appear to affect how the web services work, but the following 
 warnings are logged:
 {noformat}
 WARNING: The following warnings have been detected with resource and/or 
 provider
  classes:
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.server.applicati
 onhistoryservice.webapp.ATSWebServices$AboutInfo 
 org.apache.hadoop.yarn.server.a
 pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
 letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, 
 is t
 reated as a resource method
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.api.records.appt
 imeline.ATSPutErrors 
 org.apache.hadoop.yarn.server.applicationhistoryservice.web
 app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
 let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
 ntities), with URI template, /, is treated as a resource method
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.

[
https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898677#comment-13898677
]

Carlo Curino commented on YARN-1051:

Correct... You got the flow right.

A couple more notes for clarity:

For flexibility we are decoupling the creation of a reservation from what
applications are ran into it. I could imagine to have something like Oozie
to submit a reservation request (expressing the skyline of resource it will
need for a pipeline of jobs), get back a session id (practically a queue name),
and then submit the various jobs to it. Dynamically the Plan follower will
ensure the queue exists,and has whatever capacity the admission control
dedicated to it in every moment in time.

We plan to handle sudden collapses in cluster capacity (rack gone bad), by
reconsidering the plan as a whole (for now simple greedy replanner, deciding
what sessions to kill/reposition). This is to be able to express reservations
in absolute terms (100 containers),
instead of relative (10% of capacity)... this is particularly important for
gang jobs like Giraph that cannot deal with partial allocations well.

YARN Admission Control/Planner: enhancing the resource allocation model with
time.
--

Key: YARN-1051
URL: https://issues.apache.org/jira/browse/YARN-1051
Project: Hadoop YARN
Issue Type: Improvement
Components: capacityscheduler, resourcemanager, scheduler
Reporter: Carlo Curino
Assignee: Carlo Curino
Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf

In this umbrella JIRA we propose to extend the YARN RM to handle time
explicitly, allowing users to reserve capacity over time. This is an
important step towards SLAs, long-running services, workflows, and helps for
gang scheduling.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues


[ 
https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898687#comment-13898687
 ] 

Hudson commented on YARN-1498:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5150/])
Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in 
CHANGES.txt (sandy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


 Common scheduler changes for moving apps between queues
 ---

 Key: YARN-1498
 URL: https://issues.apache.org/jira/browse/YARN-1498
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.2.0
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 3.0.0

 Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, 
 YARN-1498.patch, YARN-1498.patch


 This JIRA is to track changes that aren't in particular schedulers but that 
 help them support moving apps between queues.  In particular, it makes sure 
 that QueueMetrics are properly updated when an app changes queue.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues


[ 
https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898683#comment-13898683
 ] 

Hudson commented on YARN-1499:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5150/])
Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in 
CHANGES.txt (sandy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


 Fair Scheduler changes for moving apps between queues
 -

 Key: YARN-1499
 URL: https://issues.apache.org/jira/browse/YARN-1499
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: scheduler
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 3.0.0

 Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues


[ 
https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898685#comment-13898685
 ] 

Hudson commented on YARN-1496:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5150/])
Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in 
CHANGES.txt (sandy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


 Protocol additions to allow moving apps between queues
 --

 Key: YARN-1496
 URL: https://issues.apache.org/jira/browse/YARN-1496
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: scheduler
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 3.0.0

 Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, 
 YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1497) Expose moving apps between queues on the command line


[ 
https://issues.apache.org/jira/browse/YARN-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898686#comment-13898686
 ] 

Hudson commented on YARN-1497:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5150/])
Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in 
CHANGES.txt (sandy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


 Expose moving apps between queues on the command line
 -

 Key: YARN-1497
 URL: https://issues.apache.org/jira/browse/YARN-1497
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: client
Affects Versions: 2.2.0
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 3.0.0

 Attachments: YARN-1497-1.patch, YARN-1497.patch, YARN-1497.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1504) RM changes for moving apps between queues


[ 
https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898684#comment-13898684
 ] 

Hudson commented on YARN-1504:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5150/])
Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in 
CHANGES.txt (sandy: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt


 RM changes for moving apps between queues
 -

 Key: YARN-1504
 URL: https://issues.apache.org/jira/browse/YARN-1504
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.2.0
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 3.0.0

 Attachments: YARN-1504-1.patch, YARN-1504.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1717) Misc improvements to leveldb timeline store


[ 
https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898703#comment-13898703
 ] 

Hadoop QA commented on YARN-1717:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12628406/YARN-1717.2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3079//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3079//console

This message is automatically generated.

 Misc improvements to leveldb timeline store
 ---

 Key: YARN-1717
 URL: https://issues.apache.org/jira/browse/YARN-1717
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1717.1.patch, YARN-1717.2.patch


 The leveldb timeline store implementation needs the following:
 * better documentation of its internal structures
 * braces for all control flow statements
 * simple locking to prevent issues related to concurrent writes
 * internal changes to enable deleting entities
 ** never overwrite existing primary filter entries
 ** add hidden reverse pointers to related entities



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings


[ 
https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898716#comment-13898716
 ] 

Hudson commented on YARN-1719:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5151 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5151/])
YARN-1719. Fixed the root path related Jersey warnings produced in 
ATSWebServices. Contributed by Billie Rinaldi. (zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567498)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java


 ATSWebServices produces jersey warnings
 ---

 Key: YARN-1719
 URL: https://issues.apache.org/jira/browse/YARN-1719
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Fix For: 2.4.0

 Attachments: YARN-1719.1.patch


 These don't appear to affect how the web services work, but the following 
 warnings are logged:
 {noformat}
 WARNING: The following warnings have been detected with resource and/or 
 provider
  classes:
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.server.applicati
 onhistoryservice.webapp.ATSWebServices$AboutInfo 
 org.apache.hadoop.yarn.server.a
 pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
 letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, 
 is t
 reated as a resource method
   WARNING: A sub-resource method, public 
 org.apache.hadoop.yarn.api.records.appt
 imeline.ATSPutErrors 
 org.apache.hadoop.yarn.server.applicationhistoryservice.web
 app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
 let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
 ntities), with URI template, /, is treated as a resource method
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN


 [ 
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-1553:
--

Target Version/s: 2.4.0

 Do not use HttpConfig.isSecure() in YARN
 

 Key: YARN-1553
 URL: https://issues.apache.org/jira/browse/YARN-1553
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: YARN-1553.000.patch, YARN-1553.001.patch, 
 YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch, 
 YARN-1553.005.patch, YARN-1553.006.patch


 HDFS-5305 and related jira decide that each individual project will have 
 their own configuration on http policy. {{HttpConfig.isSecure}} is a global 
 static method which does not fit the design anymore. The same functionality 
 should be moved into the YARN code base.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1553) Do not use HttpConfig.isSecure() in YARN

[
https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898774#comment-13898774
]

Hadoop QA commented on YARN-1553:
-

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12628392/YARN-1553.006.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy:

org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.yarn.client.TestRMFailover

The following test timeouts occurred in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

org.apache.hadoop.yarn.client.api.impl.TestNMClient

{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3078//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3078//console

This message is automatically generated.

Do not use HttpConfig.isSecure() in YARN

Key: YARN-1553
URL: https://issues.apache.org/jira/browse/YARN-1553
Project: Hadoop YARN
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: YARN-1553.000.patch, YARN-1553.001.patch,
YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch,
YARN-1553.005.patch, YARN-1553.006.patch

HDFS-5305 and related jira decide that each individual project will have
their own configuration on http policy. {{HttpConfig.isSecure}} is a global
static method which does not fit the design anymore. The same functionality
should be moved into the YARN code base.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1641) ZK store should attempt a write periodically to ensure it is still Active