[jira] [Commented] (YARN-1637) Implement a client library for java users to post entities+events
[ https://issues.apache.org/jira/browse/YARN-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897741#comment-13897741 ] Hudson commented on YARN-1637: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/478/]) YARN-1637. Implemented a client library for Java users to post timeline entities and events. Contributed by Zhijie Shen. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566752) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java Implement a client library for java users to post entities+events - Key: YARN-1637 URL: https://issues.apache.org/jira/browse/YARN-1637 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1637.1.patch, YARN-1637.2.patch, YARN-1637.3.patch, YARN-1637.4.patch, YARN-1637.5.patch, YARN-1637.6.patch, YARN-1637.7.patch, YARN-1637.8.patch This is a wrapper around the web-service to facilitate easy posting of entity+event data to the time-line server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1698) Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default
[ https://issues.apache.org/jira/browse/YARN-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897743#comment-13897743 ] Hudson commented on YARN-1698: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/478/]) YARN-1698. Fixed default TimelineStore in code to match what is documented in yarn-default.xml. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566937) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default -- Key: YARN-1698 URL: https://issues.apache.org/jira/browse/YARN-1698 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1698.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1459) RM services should depend on ConfigurationProvider during startup too
[ https://issues.apache.org/jira/browse/YARN-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897747#comment-13897747 ] Hudson commented on YARN-1459: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/478/]) YARN-1459. Changed ResourceManager to depend its service initialization on the configuration-provider mechanism during startup too. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566791) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProviderFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/FileSystemBasedConfigurationProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/LocalConfigurationProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java RM services should depend on ConfigurationProvider during startup too - Key: YARN-1459 URL: https://issues.apache.org/jira/browse/YARN-1459 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.4.0 Attachments: YARN-1459.1.patch, YARN-1459.2.patch, YARN-1459.3.patch, YARN-1459.4.patch, YARN-1459.4.patch YARN-1667, YARN-1668, YARN-1669 already changed RM to depend on a configuration provider so as to be able to refresh many configuration files across RM fail-over. The dependency on the configuration-provider by the RM should happen at its boot up time too. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1706) Create an utility function to dump timeline records to json
[ https://issues.apache.org/jira/browse/YARN-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897745#comment-13897745 ] Hudson commented on YARN-1706: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #478 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/478/]) YARN-1706. Created an utility method to dump timeline records to JSON strings. Contributed by Zhijie Shen. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566982) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/TimelineUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java Create an utility function to dump timeline records to json Key: YARN-1706 URL: https://issues.apache.org/jira/browse/YARN-1706 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1706.1.patch, YARN-1706.2.patch, YARN-1706.3.patch, YARN-1706.4.patch For verification and log purpose -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1702) Expose kill app functionality as part of RM web services
[ https://issues.apache.org/jira/browse/YARN-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reassigned YARN-1702: --- Assignee: Varun Vasudev Expose kill app functionality as part of RM web services Key: YARN-1702 URL: https://issues.apache.org/jira/browse/YARN-1702 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Vasudev Assignee: Varun Vasudev Expose functionality to kill an app via the ResourceManager web services API. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1637) Implement a client library for java users to post entities+events
[ https://issues.apache.org/jira/browse/YARN-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897840#comment-13897840 ] Hudson commented on YARN-1637: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/]) YARN-1637. Implemented a client library for Java users to post timeline entities and events. Contributed by Zhijie Shen. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566752) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java Implement a client library for java users to post entities+events - Key: YARN-1637 URL: https://issues.apache.org/jira/browse/YARN-1637 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1637.1.patch, YARN-1637.2.patch, YARN-1637.3.patch, YARN-1637.4.patch, YARN-1637.5.patch, YARN-1637.6.patch, YARN-1637.7.patch, YARN-1637.8.patch This is a wrapper around the web-service to facilitate easy posting of entity+event data to the time-line server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1459) RM services should depend on ConfigurationProvider during startup too
[ https://issues.apache.org/jira/browse/YARN-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897846#comment-13897846 ] Hudson commented on YARN-1459: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/]) YARN-1459. Changed ResourceManager to depend its service initialization on the configuration-provider mechanism during startup too. Contributed by Xuan Gong. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566791) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/ConfigurationProviderFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/FileSystemBasedConfigurationProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/LocalConfigurationProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/authorize/RMPolicyProvider.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java RM services should depend on ConfigurationProvider during startup too - Key: YARN-1459 URL: https://issues.apache.org/jira/browse/YARN-1459 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Xuan Gong Fix For: 2.4.0 Attachments: YARN-1459.1.patch, YARN-1459.2.patch, YARN-1459.3.patch, YARN-1459.4.patch, YARN-1459.4.patch YARN-1667, YARN-1668, YARN-1669 already changed RM to depend on a configuration provider so as to be able to refresh many configuration files across RM fail-over. The dependency on the configuration-provider by the RM should happen at its boot up time too. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1706) Create an utility function to dump timeline records to json
[ https://issues.apache.org/jira/browse/YARN-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897844#comment-13897844 ] Hudson commented on YARN-1706: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/]) YARN-1706. Created an utility method to dump timeline records to JSON strings. Contributed by Zhijie Shen. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566982) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/TimelineUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java Create an utility function to dump timeline records to json Key: YARN-1706 URL: https://issues.apache.org/jira/browse/YARN-1706 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1706.1.patch, YARN-1706.2.patch, YARN-1706.3.patch, YARN-1706.4.patch For verification and log purpose -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1698) Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default
[ https://issues.apache.org/jira/browse/YARN-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897842#comment-13897842 ] Hudson commented on YARN-1698: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1670 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1670/]) YARN-1698. Fixed default TimelineStore in code to match what is documented in yarn-default.xml. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566937) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default -- Key: YARN-1698 URL: https://issues.apache.org/jira/browse/YARN-1698 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1698.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1706) Create an utility function to dump timeline records to json
[ https://issues.apache.org/jira/browse/YARN-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897889#comment-13897889 ] Hudson commented on YARN-1706: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/]) YARN-1706. Created an utility method to dump timeline records to JSON strings. Contributed by Zhijie Shen. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566982) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/TimelineUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/apptimeline/TestApplicationTimelineRecords.java Create an utility function to dump timeline records to json Key: YARN-1706 URL: https://issues.apache.org/jira/browse/YARN-1706 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1706.1.patch, YARN-1706.2.patch, YARN-1706.3.patch, YARN-1706.4.patch For verification and log purpose -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1698) Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default
[ https://issues.apache.org/jira/browse/YARN-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897887#comment-13897887 ] Hudson commented on YARN-1698: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/]) YARN-1698. Fixed default TimelineStore in code to match what is documented in yarn-default.xml. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566937) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default -- Key: YARN-1698 URL: https://issues.apache.org/jira/browse/YARN-1698 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1698.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1637) Implement a client library for java users to post entities+events
[ https://issues.apache.org/jira/browse/YARN-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897885#comment-13897885 ] Hudson commented on YARN-1637: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1695 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1695/]) YARN-1637. Implemented a client library for Java users to post timeline entities and events. Contributed by Zhijie Shen. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1566752) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java Implement a client library for java users to post entities+events - Key: YARN-1637 URL: https://issues.apache.org/jira/browse/YARN-1637 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Fix For: 2.4.0 Attachments: YARN-1637.1.patch, YARN-1637.2.patch, YARN-1637.3.patch, YARN-1637.4.patch, YARN-1637.5.patch, YARN-1637.6.patch, YARN-1637.7.patch, YARN-1637.8.patch This is a wrapper around the web-service to facilitate easy posting of entity+event data to the time-line server. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1713) Implement getnewapplication as part of RM web service
[ https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-1713: Attachment: yarn-1713.patch Implement getnewapplication as part of RM web service - Key: YARN-1713 URL: https://issues.apache.org/jira/browse/YARN-1713 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: yarn-1713.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1713) Implement getnewapplication as part of RM web service
Varun Vasudev created YARN-1713: --- Summary: Implement getnewapplication as part of RM web service Key: YARN-1713 URL: https://issues.apache.org/jira/browse/YARN-1713 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: yarn-1713.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898052#comment-13898052 ] Alejandro Abdelnur commented on YARN-1695: -- err, [~vinodkv], isn't this a duplicate of the JIRA you opened back in DEC, YARN-1538? Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-191) Enhancements to YARN for Windows Server and Windows Azure development and runtime environments
[ https://issues.apache.org/jira/browse/YARN-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved YARN-191. Resolution: Fixed We've completed the intended scope of this issue, so I'm resolving it. Thank you to all contributors! Enhancements to YARN for Windows Server and Windows Azure development and runtime environments -- Key: YARN-191 URL: https://issues.apache.org/jira/browse/YARN-191 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898089#comment-13898089 ] Vinod Kumar Vavilapalli commented on YARN-1695: --- Ah, thanks. I did search for existing tickets but couldn't quite locate it using JIRA search. Closing one of these as a duplicate. Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1538) Support web-services for application-submission and AM protocols
[ https://issues.apache.org/jira/browse/YARN-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved YARN-1538. --- Resolution: Duplicate Closing this as a duplicate of a newer ticket with already created sub-tasks. Support web-services for application-submission and AM protocols Key: YARN-1538 URL: https://issues.apache.org/jira/browse/YARN-1538 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli We already have read-only web-services for YARN - MAPREDUCE-2863. It'll be great to support APIs for application submission and the AM protocols - scheduling and container-management - as alternatives to RPCs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1714) Per user view in YARN RM
Siqi Li created YARN-1714: - Summary: Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Priority: Critical ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1714) Per user view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1714: -- Attachment: YARN-1714.v1.patch Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1715) Per queue view in RM is not implemented correctly
Siqi Li created YARN-1715: - Summary: Per queue view in RM is not implemented correctly Key: YARN-1715 URL: https://issues.apache.org/jira/browse/YARN-1715 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1715) Per queue view in RM is not implemented correctly
[ https://issues.apache.org/jira/browse/YARN-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li reassigned YARN-1715: - Assignee: Siqi Li Per queue view in RM is not implemented correctly - Key: YARN-1715 URL: https://issues.apache.org/jira/browse/YARN-1715 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1715) Per queue view in RM is not implemented correctly
[ https://issues.apache.org/jira/browse/YARN-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1715: -- Description: For now, per queue view in YARN RM has not yet implemented. in RmController.java it only set page title for per queue page Per queue view in RM is not implemented correctly - Key: YARN-1715 URL: https://issues.apache.org/jira/browse/YARN-1715 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li For now, per queue view in YARN RM has not yet implemented. in RmController.java it only set page title for per queue page -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898112#comment-13898112 ] Alejandro Abdelnur commented on YARN-1695: -- Before we start coding work on this, it would be great to see how security will handled (authentication, ACLs, tokens, etc). I'm a bit a concern about introducing a second protocol everywhere. From a maintenance and security risk point of view is doubling our development/support efforts. Granted, HDFS offers data over RPC and HTTP. But, HTTP, when using HttpFS (how I recommend using it HTTP access) is a gateway that ends up doing RPC to HDFS. Thus the only protocol accessing HDFS is RPC. Have we considered having a C implementation of Hadoop RPC, with the multiplatform support of protobuffers that may give us the multi-platform support we are aiming to with a single protocol interface. Thoughts? Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898126#comment-13898126 ] Hadoop QA commented on YARN-1714: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628275/YARN-1714.v1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3066//console This message is automatically generated. Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1714) Per user view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1714: -- Attachment: (was: YARN-1714.v1.patch) Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1714) Per user view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1714: -- Attachment: YARN-1714.v1.patch Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898146#comment-13898146 ] Hadoop QA commented on YARN-1714: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628283/YARN-1714.v1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3067//console This message is automatically generated. Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1714) Per user view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1714: -- Attachment: YARN-1714.v2.patch Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1202) Verify RM HA works in secure clusters
[ https://issues.apache.org/jira/browse/YARN-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1202: --- Assignee: Wing Yew Poon (was: Karthik Kambatla) Verify RM HA works in secure clusters - Key: YARN-1202 URL: https://issues.apache.org/jira/browse/YARN-1202 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Karthik Kambatla Assignee: Wing Yew Poon We should verify RM HA works in secure clusters, and fix any issues that we come across. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1202) Verify RM HA works in secure clusters
[ https://issues.apache.org/jira/browse/YARN-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898176#comment-13898176 ] Karthik Kambatla commented on YARN-1202: [~wypoon] has been doing a lot of RM HA testing on both secure and non-secure clusters. Assigning this to him. Wing Yew - can you report what you found so far? Verify RM HA works in secure clusters - Key: YARN-1202 URL: https://issues.apache.org/jira/browse/YARN-1202 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Karthik Kambatla Assignee: Wing Yew Poon We should verify RM HA works in secure clusters, and fix any issues that we come across. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1687) Refactoring timeline classes to remove app related words
[ https://issues.apache.org/jira/browse/YARN-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1687: -- Attachment: YARN-1687.1.patch Created a patch that refactors the timeline module: class names, package names, javadoc, variable names, configuration names, webservices paths. Refactoring timeline classes to remove app related words -- Key: YARN-1687 URL: https://issues.apache.org/jira/browse/YARN-1687 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: YARN-1687.1.patch Remove ATS prefix, change package name, fix javadoc and so on -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-1710: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-1051 Admission Control: agents to allocate reservation - Key: YARN-1710 URL: https://issues.apache.org/jira/browse/YARN-1710 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Carlo Curino This JIRA tracks the algorithms used to allocate a user ReservationRequest coming in from the new reservation API (YARN-1708), in the inventory subsystem (YARN-1709) maintaining the current plan for the cluster. The focus of this agents is to quickly find a solution for the set of contraints provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1707) Making the CapacityScheduler more dynamic
[ https://issues.apache.org/jira/browse/YARN-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-1707: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-1051 Making the CapacityScheduler more dynamic - Key: YARN-1707 URL: https://issues.apache.org/jira/browse/YARN-1707 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Carlo Curino Assignee: Carlo Curino The CapacityScheduler is a rather static at the moment, and refreshqueue provides a rather heavy-handed way to reconfigure it. Moving towards long-running services (tracked in YARN-896) and to enable more advanced admission control and resource parcelling we need to make the CapacityScheduler more dynamic. This is instrumental to the umbrella jira YARN-1051. Concretely this require the following changes: * create queues dynamically * destroy queues dynamically * dynamically change queue parameters (e.g., capacity) * modify refreshqueue validation to enforce sum(child.getCapacity())= 100% instead of ==100% We limit this to LeafQueues. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1708) Add a public API to reserve resources (part of YARN-1051)
[ https://issues.apache.org/jira/browse/YARN-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-1708: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-1051 Add a public API to reserve resources (part of YARN-1051) - Key: YARN-1708 URL: https://issues.apache.org/jira/browse/YARN-1708 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Carlo Curino This JIRA tracks the definition of a new public API for YARN, which allows users to reserve resources (think of time-bounded queues). This is part of the admission control enhancement proposed in YARN-1051. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1709) Admission Control: inventory subsystem
[ https://issues.apache.org/jira/browse/YARN-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-1709: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-1051 Admission Control: inventory subsystem -- Key: YARN-1709 URL: https://issues.apache.org/jira/browse/YARN-1709 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Carlo Curino This JIRA is about the key data structure used to track resources over time to enable YARN-1051. The inventory subsystem is conceptually a plan of how the capacity scheduler will be configured over-time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-1711: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-1051 CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1712) Admission Control: plan follower
[ https://issues.apache.org/jira/browse/YARN-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-1712: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-1051 Admission Control: plan follower Key: YARN-1712 URL: https://issues.apache.org/jira/browse/YARN-1712 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler, resourcemanager Reporter: Carlo Curino Assignee: Carlo Curino This JIRA tracks a thread that continuously propagates the current state of an inventory subsystem to the scheduler. As the inventory subsystem store the plan of how the resources should be subdivided, the work we propose in this JIRA realizes such plan by dynamically instructing the CapacityScheduler to add/remove/resize queues to follow the plan. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1708) Add a public API to reserve resources (part of YARN-1051)
[ https://issues.apache.org/jira/browse/YARN-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino reassigned YARN-1708: -- Assignee: Subramaniam Krishnan Add a public API to reserve resources (part of YARN-1051) - Key: YARN-1708 URL: https://issues.apache.org/jira/browse/YARN-1708 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Carlo Curino Assignee: Subramaniam Krishnan This JIRA tracks the definition of a new public API for YARN, which allows users to reserve resources (think of time-bounded queues). This is part of the admission control enhancement proposed in YARN-1051. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1709) Admission Control: inventory subsystem
[ https://issues.apache.org/jira/browse/YARN-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subramaniam Krishnan reassigned YARN-1709: -- Assignee: Subramaniam Krishnan Admission Control: inventory subsystem -- Key: YARN-1709 URL: https://issues.apache.org/jira/browse/YARN-1709 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Carlo Curino Assignee: Subramaniam Krishnan This JIRA is about the key data structure used to track resources over time to enable YARN-1051. The inventory subsystem is conceptually a plan of how the capacity scheduler will be configured over-time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1710) Admission Control: agents to allocate reservation
[ https://issues.apache.org/jira/browse/YARN-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subramaniam Krishnan reassigned YARN-1710: -- Assignee: Subramaniam Krishnan Admission Control: agents to allocate reservation - Key: YARN-1710 URL: https://issues.apache.org/jira/browse/YARN-1710 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Carlo Curino Assignee: Subramaniam Krishnan This JIRA tracks the algorithms used to allocate a user ReservationRequest coming in from the new reservation API (YARN-1708), in the inventory subsystem (YARN-1709) maintaining the current plan for the cluster. The focus of this agents is to quickly find a solution for the set of contraints provided by the user, and the physical constraints of the plan. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
[ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subramaniam Krishnan updated YARN-1051: --- Attachment: YARN-1051-design.pdf Attaching the approach doc that describes the overall intent for interested readers. The doc also lists the breakdown into incremental sub-tasks.Any suggestions/thoughts are welcome, we will incorporate feedback as it comes in. YARN Admission Control/Planner: enhancing the resource allocation model with time. -- Key: YARN-1051 URL: https://issues.apache.org/jira/browse/YARN-1051 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler Reporter: Carlo Curino Assignee: Carlo Curino Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf In this umbrella JIRA we propose to extend the YARN RM to handle time explicitly, allowing users to reserve capacity over time. This is an important step towards SLAs, long-running services, workflows, and helps for gang scheduling. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1697) NodeManager reports negative running containers
[ https://issues.apache.org/jira/browse/YARN-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898309#comment-13898309 ] Hudson commented on YARN-1697: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5148 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5148/]) YARN-1697. NodeManager reports negative running containers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567356) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java NodeManager reports negative running containers --- Key: YARN-1697 URL: https://issues.apache.org/jira/browse/YARN-1697 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.3.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: YARN-1697.patch We're seeing the NodeManager metrics report a negative number of running containers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898314#comment-13898314 ] Hadoop QA commented on YARN-1714: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628295/YARN-1714.v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3068//console This message is automatically generated. Per user view in YARN RM Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1690) sending ATS events from Distributed shell
[ https://issues.apache.org/jira/browse/YARN-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898329#comment-13898329 ] Zhijie Shen commented on YARN-1690: --- TimelineClient is ready. The patch is stale now. sending ATS events from Distributed shell -- Key: YARN-1690 URL: https://issues.apache.org/jira/browse/YARN-1690 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: YARN-1690-1.patch, YARN-1690-2.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1676) Make admin refreshUserToGroupsMappings of configuration work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1676: Attachment: YARN-1676.4.patch Modify the testcase to call updateGroups() to update groups before calling refreshUserToGroupsMappings. After that, verify whether the groups are updated. Make admin refreshUserToGroupsMappings of configuration work across RM failover --- Key: YARN-1676 URL: https://issues.apache.org/jira/browse/YARN-1676 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1667.3.patch, YARN-1676.1.patch, YARN-1676.2.patch, YARN-1676.3.patch, YARN-1676.4.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (YARN-1682) TestRMRestart#testRMRestartSucceededApp occasionally fails
[ https://issues.apache.org/jira/browse/YARN-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved YARN-1682. -- Resolution: Cannot Reproduce TestRMRestart#testRMRestartSucceededApp occasionally fails -- Key: YARN-1682 URL: https://issues.apache.org/jira/browse/YARN-1682 Project: Hadoop YARN Issue Type: Test Reporter: Ted Yu From https://builds.apache.org/job/Hadoop-Yarn-trunk/468/console : {code} testRMRestartSucceededApp(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart) Time elapsed: 8.129 sec FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.verifyAppReportAfterRMRestart(TestRMRestart.java:900) at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartSucceededApp(TestRMRestart.java:774) {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1515) Ability to dump the container threads and stop the containers in a single RPC
[ https://issues.apache.org/jira/browse/YARN-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898425#comment-13898425 ] Karthik Kambatla commented on YARN-1515: I just barely skimmed through the patch. Before taking a closer look at it, can we consolidate the new ContainerManagementProtocol#stopContainers1 and the existing ContainerManagementProtocol#stopContainers? That would probably make the patch shorter too - easier to review :) Ability to dump the container threads and stop the containers in a single RPC - Key: YARN-1515 URL: https://issues.apache.org/jira/browse/YARN-1515 Project: Hadoop YARN Issue Type: New Feature Components: api, nodemanager Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: YARN-1515.v01.patch, YARN-1515.v02.patch This is needed to implement MAPREDUCE-5044 to enable thread diagnostics for timed-out task attempts. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[ https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898433#comment-13898433 ] Ted Yu commented on YARN-1296: -- Is more review needed ? schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper -- Key: YARN-1296 URL: https://issues.apache.org/jira/browse/YARN-1296 Project: Hadoop YARN Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: yarn-1296-v1.patch Here is related code: {code} public Allocation allocate(ApplicationAttemptId attemptId, ListResourceRequest resourceRequests, ListContainerId containerIds, ListString strings, ListString strings2) { if (metricsON) { final Timer.Context context = schedulerAllocateTimer.time(); {code} samplerLock should be used to guard the access. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1583) Ineffective state check in FairSchedulerAppsBlock#render()
[ https://issues.apache.org/jira/browse/YARN-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated YARN-1583: - Priority: Major (was: Minor) Ineffective state check in FairSchedulerAppsBlock#render() -- Key: YARN-1583 URL: https://issues.apache.org/jira/browse/YARN-1583 Project: Hadoop YARN Issue Type: Bug Reporter: Ted Yu Starting line 90: {code} for (RMApp app : apps.values()) { if (reqAppStates != null !reqAppStates.contains(app.getState())) { {code} reqAppStates is of type YarnApplicationState. app.getState() returns RMAppState. These are two different enum types. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Moved] (YARN-1716) finishedStatus is accessed without proper locking in RMContainerImpl#FinishedTransition#transition()
[ https://issues.apache.org/jira/browse/YARN-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu moved MAPREDUCE-5753 to YARN-1716: - Key: YARN-1716 (was: MAPREDUCE-5753) Project: Hadoop YARN (was: Hadoop Map/Reduce) finishedStatus is accessed without proper locking in RMContainerImpl#FinishedTransition#transition() Key: YARN-1716 URL: https://issues.apache.org/jira/browse/YARN-1716 Project: Hadoop YARN Issue Type: Bug Reporter: Ted Yu {code} container.finishedStatus = finishedEvent.getRemoteContainerStatus(); {code} The access should be guarded by obtaining readLock -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1714: -- Summary: Per user and per queue view in YARN RM (was: Per user view in YARN RM) Per user and per queue view in YARN RM -- Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cindy Li updated YARN-1525: --- Attachment: YARN1525.v9.patch Xuan and Karthik, I've included changes according to your comments: added security enabling check, minimized format related changes, changed comments for test case. Thanks. Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1714: -- Description: ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Provide rest style url to access only user specified queue or user apps. For instance, http://hadoop-example.com:50030/cluster/user/toto displays apps owned by toto http://hadoop-example.com:50030/cluster/user/toto,glinda displays apps owned by toto and glinda http://hadoop-example.com:50030/cluster/queue/root.queue1 displays apps in root.queue1 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 displays apps in root.queue1 and root.queue2 was:ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Per user and per queue view in YARN RM -- Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Provide rest style url to access only user specified queue or user apps. For instance, http://hadoop-example.com:50030/cluster/user/toto displays apps owned by toto http://hadoop-example.com:50030/cluster/user/toto,glinda displays apps owned by toto and glinda http://hadoop-example.com:50030/cluster/queue/root.queue1 displays apps in root.queue1 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1296) schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper
[ https://issues.apache.org/jira/browse/YARN-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898475#comment-13898475 ] Hadoop QA commented on YARN-1296: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607939/yarn-1296-v1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-sls. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3070//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3070//console This message is automatically generated. schedulerAllocateTimer is accessed without holding samplerLock in ResourceSchedulerWrapper -- Key: YARN-1296 URL: https://issues.apache.org/jira/browse/YARN-1296 Project: Hadoop YARN Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: yarn-1296-v1.patch Here is related code: {code} public Allocation allocate(ApplicationAttemptId attemptId, ListResourceRequest resourceRequests, ListContainerId containerIds, ListString strings, ListString strings2) { if (metricsON) { final Timer.Context context = schedulerAllocateTimer.time(); {code} samplerLock should be used to guard the access. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
[ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898478#comment-13898478 ] Karthik Kambatla commented on YARN-1051: Thanks Carlo and Subru for sharing the TR and design-doc. Can you please verify my understanding is right (haven't read the TR yet, just the design doc): The {{Admission Control}} box (Planning agent and Inventory) takes care of figuring out if a reservation is feasible and managing the corresponding inventory. Plan Follower would take care of actually submitting the applications to the RM (scheduler) and scheduler-queue configurations come show time; the updated scheduler-queue configurations would ensure these apps would actually get the resources they need. YARN Admission Control/Planner: enhancing the resource allocation model with time. -- Key: YARN-1051 URL: https://issues.apache.org/jira/browse/YARN-1051 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler Reporter: Carlo Curino Assignee: Carlo Curino Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf In this umbrella JIRA we propose to extend the YARN RM to handle time explicitly, allowing users to reserve capacity over time. This is an important step towards SLAs, long-running services, workflows, and helps for gang scheduling. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898487#comment-13898487 ] Karthik Kambatla commented on YARN-1525: Nits before actual review: # webapp/Dispatcher.java and webapp/Dispatcher.java have spurious changes to the license header - may be IDE induced. # The methods in RMHAUtils can be moved to HAUtil. It would be nice to get this done soon, so these are also taken care of by Vinod's attempt to clean up HAUtil. Sorry for the additional trouble, Vinod. Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1588: -- Attachment: YARN-1588.1.patch Rebind NM tokens for previous attempt's running containers to the new attempt - Key: YARN-1588 URL: https://issues.apache.org/jira/browse/YARN-1588 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Attachments: YARN-1588.1.patch, YARN-1588.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898488#comment-13898488 ] Karthik Kambatla commented on YARN-1525: Oh - missed these - would be nice to not include these either: {code} -@InterfaceAudience.LimitedPrivate({YARN, MapReduce}) +@InterfaceAudience.LimitedPrivate({ YARN, MapReduce }) public abstract class WebApp extends ServletModule { private static final Logger LOG = LoggerFactory.getLogger(WebApp.class); - public enum HTTP { GET, POST, HEAD, PUT, DELETE }; + public enum HTTP { +GET, POST, HEAD, PUT, DELETE + }; {code} Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reassigned YARN-1588: - Assignee: Jian He Rebind NM tokens for previous attempt's running containers to the new attempt - Key: YARN-1588 URL: https://issues.apache.org/jira/browse/YARN-1588 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1588.1.patch, YARN-1588.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1676) Make admin refreshUserToGroupsMappings of configuration work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898496#comment-13898496 ] Hadoop QA commented on YARN-1676: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628338/YARN-1676.4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3069//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3069//console This message is automatically generated. Make admin refreshUserToGroupsMappings of configuration work across RM failover --- Key: YARN-1676 URL: https://issues.apache.org/jira/browse/YARN-1676 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1667.3.patch, YARN-1676.1.patch, YARN-1676.2.patch, YARN-1676.3.patch, YARN-1676.4.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898497#comment-13898497 ] Hadoop QA commented on YARN-1525: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628362/YARN1525.v9.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3071//console This message is automatically generated. Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1714: -- Attachment: YARN-1714.v3.patch Per user and per queue view in YARN RM -- Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, YARN-1714.v3.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Provide rest style url to access only user specified queue or user apps. For instance, http://hadoop-example.com:50030/cluster/user/toto displays apps owned by toto http://hadoop-example.com:50030/cluster/user/toto,glinda displays apps owned by toto and glinda http://hadoop-example.com:50030/cluster/queue/root.queue1 displays apps in root.queue1 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1525) Web UI should redirect to active RM when HA is enabled.
[ https://issues.apache.org/jira/browse/YARN-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898500#comment-13898500 ] Cindy Li commented on YARN-1525: @Karthik, RMHAUtils.java couldn't be merged into HAUtil.java because of project setup, as it uses RMHAServiceTarget which would not be accessible from HAUtil.java. Yes. the format are automatically changed by IDE. I can try to manually remove them. Web UI should redirect to active RM when HA is enabled. --- Key: YARN-1525 URL: https://issues.apache.org/jira/browse/YARN-1525 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Cindy Li Attachments: YARN1525.patch.v1, YARN1525.patch.v2, YARN1525.patch.v3, YARN1525.v7.patch, YARN1525.v7.patch, YARN1525.v8.patch, YARN1525.v9.patch When failover happens, web UI should redirect to the current active rm. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1717) Misc improvements to leveldb timeline store
Billie Rinaldi created YARN-1717: Summary: Misc improvements to leveldb timeline store Key: YARN-1717 URL: https://issues.apache.org/jira/browse/YARN-1717 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi The leveldb timeline store implementation needs the following: * better documentation of its internal structures * braces for all control flow statements * simple locking to prevent issues related to concurrent writes * internal changes to enable deleting entities ** never overwrite existing primary filter entries ** add hidden reverse pointers to related entities -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1492) truly shared cache for jars (jobjar/libjar)
[ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated YARN-1492: --- Attachment: shared_cache_design_v5.pdf Attached is a v5 of the design document. Some of the major changes are: # Cache management logic moved from the client to a shared cache manager and localization service. As a benefit, cached resources are now read-only and managed solely by a trusted source (SCM and NodeManager) running as its own user. # Caching of a resource is now off the job submission path and happens asynchronously. If submitting to the cache fails, it does not affect the job. # Improved robustness when dealing with clients that don’t clean up after themselves correctly. Feedback is always appreciated. I will be starting implementation shortly. truly shared cache for jars (jobjar/libjar) --- Key: YARN-1492 URL: https://issues.apache.org/jira/browse/YARN-1492 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: shared_cache_design.pdf, shared_cache_design_v2.pdf, shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, shared_cache_design_v5.pdf Currently there is the distributed cache that enables you to cache jars and files so that attempts from the same job can reuse them. However, sharing is limited with the distributed cache because it is normally on a per-job basis. On a large cluster, sometimes copying of jobjars and libjars becomes so prevalent that it consumes a large portion of the network bandwidth, not to speak of defeating the purpose of bringing compute to where data is. This is wasteful because in most cases code doesn't change much across many jobs. I'd like to propose and discuss feasibility of introducing a truly shared cache so that multiple jobs from multiple users can share and cache jars. This JIRA is to open the discussion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898512#comment-13898512 ] Hadoop QA commented on YARN-1714: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628368/YARN-1714.v3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3073//console This message is automatically generated. Per user and per queue view in YARN RM -- Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, YARN-1714.v3.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Provide rest style url to access only user specified queue or user apps. For instance, http://hadoop-example.com:50030/cluster/user/toto displays apps owned by toto http://hadoop-example.com:50030/cluster/user/toto,glinda displays apps owned by toto and glinda http://hadoop-example.com:50030/cluster/queue/root.queue1 displays apps in root.queue1 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1717) Misc improvements to leveldb timeline store
[ https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1717: - Attachment: YARN-1717.1.patch Here is my first attempt at this patch. If desired, I can remove the new methods appearing at the end of the leveldb impl that enable offline entity age off and put those on a separate ticket (i.e. getEntityTypes, removeStartTime, deleteKeysWithPrefix, deleteEntity, deleteOldEntries). Misc improvements to leveldb timeline store --- Key: YARN-1717 URL: https://issues.apache.org/jira/browse/YARN-1717 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1717.1.patch The leveldb timeline store implementation needs the following: * better documentation of its internal structures * braces for all control flow statements * simple locking to prevent issues related to concurrent writes * internal changes to enable deleting entities ** never overwrite existing primary filter entries ** add hidden reverse pointers to related entities -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898537#comment-13898537 ] Hadoop QA commented on YARN-1588: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628365/YARN-1588.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3072//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/3072//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3072//console This message is automatically generated. Rebind NM tokens for previous attempt's running containers to the new attempt - Key: YARN-1588 URL: https://issues.apache.org/jira/browse/YARN-1588 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1588.1.patch, YARN-1588.1.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules
Sandy Ryza created YARN-1718: Summary: Fix a couple isTerminals in Fair Scheduler queue placement rules Key: YARN-1718 URL: https://issues.apache.org/jira/browse/YARN-1718 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza SecondaryGroupExistingQueue and Default are incorrect -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules
[ https://issues.apache.org/jira/browse/YARN-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-1718: - Attachment: YARN-1718.patch Fix a couple isTerminals in Fair Scheduler queue placement rules - Key: YARN-1718 URL: https://issues.apache.org/jira/browse/YARN-1718 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1718.patch SecondaryGroupExistingQueue and Default are incorrect -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1588: -- Attachment: YARN-1588.2.patch Rebind NM tokens for previous attempt's running containers to the new attempt - Key: YARN-1588 URL: https://issues.apache.org/jira/browse/YARN-1588 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1719) ATSWebServices produces jersey warnings
Billie Rinaldi created YARN-1719: Summary: ATSWebServices produces jersey warnings Key: YARN-1719 URL: https://issues.apache.org/jira/browse/YARN-1719 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1719.1.patch These don't appear to affect how the web services work, but the following warnings are logged: {noformat} WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, is t reated as a resource method WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE ntities), with URI template, /, is treated as a resource method {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1719) ATSWebServices produces jersey warnings
[ https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1719: - Attachment: YARN-1719.1.patch ATSWebServices produces jersey warnings --- Key: YARN-1719 URL: https://issues.apache.org/jira/browse/YARN-1719 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1719.1.patch These don't appear to affect how the web services work, but the following warnings are logged: {noformat} WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, is t reated as a resource method WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE ntities), with URI template, /, is treated as a resource method {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1495) Allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898568#comment-13898568 ] Sandy Ryza commented on YARN-1495: -- bq. From an API point of view, there should be a way for the application at run-time/registration time to find out what features are supported or not supported by the currently configured scheduler in the RM. That makes total sense to me. bq. Now, for moving apps across schedulers - given that it is a client only feature and there is no changes required in an application, my previous comment's argument does not hold for this feature. Great. Will go forward with the merge to branch-2. Allow moving apps between queues Key: YARN-1495 URL: https://issues.apache.org/jira/browse/YARN-1495 Project: Hadoop YARN Issue Type: New Feature Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza This is an umbrella JIRA for work needed to allow moving YARN applications from one queue to another. The work will consist of additions in the command line options, additions in the client RM protocol, and changes in the schedulers to support this. I have a picture of how this should function in the Fair Scheduler, but I'm not familiar enough with the Capacity Scheduler for the same there. Ultimately, the decision to whether an application can be moved should go down to the scheduler - some schedulers may wish not to support this at all. However, schedulers that do support it should share some common semantics around ACLs and what happens to running containers. Here is how I see the general semantics working out: * A move request is issued by the client. After it gets past ACLs, the scheduler checks whether executing the move will violate any constraints. For the Fair Scheduler, these would be queue maxRunningApps and queue maxResources constraints * All running containers are transferred from the old queue to the new queue * All outstanding requests are transferred from the old queue to the new queue Here is I see the ACLs of this working out: * To move an app from a queue a user must have modify access on the app or administer access on the queue * To move an app to a queue a user must have submit access on the queue or administer access on the queue -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1676) Make admin refreshUserToGroupsMappings of configuration work across RM failover
[ https://issues.apache.org/jira/browse/YARN-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898571#comment-13898571 ] Vinod Kumar Vavilapalli commented on YARN-1676: --- +1, this looks good. Checking this in. Make admin refreshUserToGroupsMappings of configuration work across RM failover --- Key: YARN-1676 URL: https://issues.apache.org/jira/browse/YARN-1676 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Attachments: YARN-1667.3.patch, YARN-1676.1.patch, YARN-1676.2.patch, YARN-1676.3.patch, YARN-1676.4.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (YARN-1720) QueuePlacementRule.SecondaryGroupExistingQueue should not be terminal
Aditya Acharya created YARN-1720: Summary: QueuePlacementRule.SecondaryGroupExistingQueue should not be terminal Key: YARN-1720 URL: https://issues.apache.org/jira/browse/YARN-1720 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.2.0 Reporter: Aditya Acharya Assignee: Sandy Ryza The SecondaryGroupExistingQueue QueuePlacementRule's isTerminal() method should always return false, not create, because it does not, in fact, create a queue ever. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[ https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated YARN-1553: - Attachment: YARN-1553.006.patch The v6 patch ensures that the AM always starts an HTTP server. Thanks [~vinodkv] for hunting the bug. Do not use HttpConfig.isSecure() in YARN Key: YARN-1553 URL: https://issues.apache.org/jira/browse/YARN-1553 Project: Hadoop YARN Issue Type: Bug Reporter: Haohui Mai Assignee: Haohui Mai Attachments: YARN-1553.000.patch, YARN-1553.001.patch, YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch, YARN-1553.005.patch, YARN-1553.006.patch HDFS-5305 and related jira decide that each individual project will have their own configuration on http policy. {{HttpConfig.isSecure}} is a global static method which does not fit the design anymore. The same functionality should be moved into the YARN code base. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1717) Misc improvements to leveldb timeline store
[ https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898575#comment-13898575 ] Hadoop QA commented on YARN-1717: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628373/YARN-1717.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3074//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3074//console This message is automatically generated. Misc improvements to leveldb timeline store --- Key: YARN-1717 URL: https://issues.apache.org/jira/browse/YARN-1717 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1717.1.patch The leveldb timeline store implementation needs the following: * better documentation of its internal structures * braces for all control flow statements * simple locking to prevent issues related to concurrent writes * internal changes to enable deleting entities ** never overwrite existing primary filter entries ** add hidden reverse pointers to related entities -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[ https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898599#comment-13898599 ] Hadoop QA commented on YARN-1719: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628389/YARN-1719.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3076//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3076//console This message is automatically generated. ATSWebServices produces jersey warnings --- Key: YARN-1719 URL: https://issues.apache.org/jira/browse/YARN-1719 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1719.1.patch These don't appear to affect how the web services work, but the following warnings are logged: {noformat} WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, is t reated as a resource method WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE ntities), with URI template, /, is treated as a resource method {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1495) Allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898601#comment-13898601 ] Hitesh Shah commented on YARN-1495: --- [~sandyr] Just to be clear, I have not reviewed the code as such so please do not consider my comment as a +1 if that is required for the merge. I am assuming there are others who are more familiar with these changes and have reviewed/+1'ed it for the merge. Allow moving apps between queues Key: YARN-1495 URL: https://issues.apache.org/jira/browse/YARN-1495 Project: Hadoop YARN Issue Type: New Feature Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza This is an umbrella JIRA for work needed to allow moving YARN applications from one queue to another. The work will consist of additions in the command line options, additions in the client RM protocol, and changes in the schedulers to support this. I have a picture of how this should function in the Fair Scheduler, but I'm not familiar enough with the Capacity Scheduler for the same there. Ultimately, the decision to whether an application can be moved should go down to the scheduler - some schedulers may wish not to support this at all. However, schedulers that do support it should share some common semantics around ACLs and what happens to running containers. Here is how I see the general semantics working out: * A move request is issued by the client. After it gets past ACLs, the scheduler checks whether executing the move will violate any constraints. For the Fair Scheduler, these would be queue maxRunningApps and queue maxResources constraints * All running containers are transferred from the old queue to the new queue * All outstanding requests are transferred from the old queue to the new queue Here is I see the ACLs of this working out: * To move an app from a queue a user must have modify access on the app or administer access on the queue * To move an app to a queue a user must have submit access on the queue or administer access on the queue -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (YARN-1417) RM may issue expired container tokens to AM while issuing new containers.
[ https://issues.apache.org/jira/browse/YARN-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reassigned YARN-1417: - Assignee: Jian He (was: Omkar Vinit Joshi) RM may issue expired container tokens to AM while issuing new containers. - Key: YARN-1417 URL: https://issues.apache.org/jira/browse/YARN-1417 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Jian He Priority: Blocker Attachments: YARN-1417.2.patch Today we create new container token when we create container in RM as a part of schedule cycle. However that container may get reserved or assigned. If the container gets reserved and remains like that (in reserved state) for more than container token expiry interval then RM will end up issuing container with expired token. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1718) Fix a couple isTerminals in Fair Scheduler queue placement rules
[ https://issues.apache.org/jira/browse/YARN-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898628#comment-13898628 ] Hadoop QA commented on YARN-1718: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628387/YARN-1718.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3075//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3075//console This message is automatically generated. Fix a couple isTerminals in Fair Scheduler queue placement rules - Key: YARN-1718 URL: https://issues.apache.org/jira/browse/YARN-1718 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1718.patch SecondaryGroupExistingQueue and Default are incorrect -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt
[ https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898631#comment-13898631 ] Hadoop QA commented on YARN-1588: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628388/YARN-1588.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3077//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/3077//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3077//console This message is automatically generated. Rebind NM tokens for previous attempt's running containers to the new attempt - Key: YARN-1588 URL: https://issues.apache.org/jira/browse/YARN-1588 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1641) ZK store should attempt a write periodically to ensure it is still Active
[ https://issues.apache.org/jira/browse/YARN-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898633#comment-13898633 ] Sandy Ryza commented on YARN-1641: -- The added thread should be given a name. Otherwise, LGTM. ZK store should attempt a write periodically to ensure it is still Active - Key: YARN-1641 URL: https://issues.apache.org/jira/browse/YARN-1641 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: yarn-1641-1.patch Fencing in ZK store kicks in when the RM tries to write something to the store. If the RM doesn't write anything to the store, it doesn't get fenced and can continue to assume being the Active. By periodically writing a file (say, every RM_ZK_TIMEOUT_MS seconds), we can ensure it gets fenced. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1717) Misc improvements to leveldb timeline store
[ https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-1717: - Attachment: YARN-1717.2.patch Fixed findbugs warning. Misc improvements to leveldb timeline store --- Key: YARN-1717 URL: https://issues.apache.org/jira/browse/YARN-1717 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1717.1.patch, YARN-1717.2.patch The leveldb timeline store implementation needs the following: * better documentation of its internal structures * braces for all control flow statements * simple locking to prevent issues related to concurrent writes * internal changes to enable deleting entities ** never overwrite existing primary filter entries ** add hidden reverse pointers to related entities -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1495) Allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898666#comment-13898666 ] Karthik Kambatla commented on YARN-1495: None of the patches introduce any Public-Stable APIs. I think it is reasonable to merge them to branch-2. +1. Thanks Sandy. Allow moving apps between queues Key: YARN-1495 URL: https://issues.apache.org/jira/browse/YARN-1495 Project: Hadoop YARN Issue Type: New Feature Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza This is an umbrella JIRA for work needed to allow moving YARN applications from one queue to another. The work will consist of additions in the command line options, additions in the client RM protocol, and changes in the schedulers to support this. I have a picture of how this should function in the Fair Scheduler, but I'm not familiar enough with the Capacity Scheduler for the same there. Ultimately, the decision to whether an application can be moved should go down to the scheduler - some schedulers may wish not to support this at all. However, schedulers that do support it should share some common semantics around ACLs and what happens to running containers. Here is how I see the general semantics working out: * A move request is issued by the client. After it gets past ACLs, the scheduler checks whether executing the move will violate any constraints. For the Fair Scheduler, these would be queue maxRunningApps and queue maxResources constraints * All running containers are transferred from the old queue to the new queue * All outstanding requests are transferred from the old queue to the new queue Here is I see the ACLs of this working out: * To move an app from a queue a user must have modify access on the app or administer access on the queue * To move an app to a queue a user must have submit access on the queue or administer access on the queue -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1714) Per user and per queue view in YARN RM
[ https://issues.apache.org/jira/browse/YARN-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898669#comment-13898669 ] Zhijie Shen commented on YARN-1714: --- Hi Siqi, I've some thoughts about this issue: 1. Actually the current apps page provide users the full text search over the app list. Therefore, you can input user/queue to retrieve the sub list of a certain user/queue. Though I understand it's not perfect way as a complete list has to be retrieved anyway, it should meet your basic requirement, shouldn't it? Or you're looking for some thing efficient? 2. Another consideration is that now REST API is providing more filters (including user/queue) than web UI to get applications. Ideally, the two interfaces should have same or equivalent capability of exposing data. Probably you want to think about other filters here as well. 3. I had a quick look at the patch. I didn't see there's an iterative way for users to do the query other than editing url. Did I miss something? 4. Should we reuse AppsBlock? Per user and per queue view in YARN RM -- Key: YARN-1714 URL: https://issues.apache.org/jira/browse/YARN-1714 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: YARN-1714.v1.patch, YARN-1714.v2.patch, YARN-1714.v3.patch ResourceManager exposes either one or all jobs via WebUI. It would be good to have filter for user so that see only their jobs. Provide rest style url to access only user specified queue or user apps. For instance, http://hadoop-example.com:50030/cluster/user/toto displays apps owned by toto http://hadoop-example.com:50030/cluster/user/toto,glinda displays apps owned by toto and glinda http://hadoop-example.com:50030/cluster/queue/root.queue1 displays apps in root.queue1 http://hadoop-example.com:50030/cluster/queue/root.queue1,root.queue2 displays apps in root.queue1 and root.queue2 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[ https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898673#comment-13898673 ] Zhijie Shen commented on YARN-1719: --- +1. Verified locally, the warnings were gone. Will commit it. ATSWebServices produces jersey warnings --- Key: YARN-1719 URL: https://issues.apache.org/jira/browse/YARN-1719 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1719.1.patch These don't appear to affect how the web services work, but the following warnings are logged: {noformat} WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, is t reated as a resource method WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE ntities), with URI template, /, is treated as a resource method {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
[ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898677#comment-13898677 ] Carlo Curino commented on YARN-1051: Correct... You got the flow right. A couple more notes for clarity: For flexibility we are decoupling the creation of a reservation from what applications are ran into it. I could imagine to have something like Oozie to submit a reservation request (expressing the skyline of resource it will need for a pipeline of jobs), get back a session id (practically a queue name), and then submit the various jobs to it. Dynamically the Plan follower will ensure the queue exists,and has whatever capacity the admission control dedicated to it in every moment in time. We plan to handle sudden collapses in cluster capacity (rack gone bad), by reconsidering the plan as a whole (for now simple greedy replanner, deciding what sessions to kill/reposition). This is to be able to express reservations in absolute terms (100 containers), instead of relative (10% of capacity)... this is particularly important for gang jobs like Giraph that cannot deal with partial allocations well. YARN Admission Control/Planner: enhancing the resource allocation model with time. -- Key: YARN-1051 URL: https://issues.apache.org/jira/browse/YARN-1051 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager, scheduler Reporter: Carlo Curino Assignee: Carlo Curino Attachments: YARN-1051-design.pdf, curino_MSR-TR-2013-108.pdf In this umbrella JIRA we propose to extend the YARN RM to handle time explicitly, allowing users to reserve capacity over time. This is an important step towards SLAs, long-running services, workflows, and helps for gang scheduling. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1498) Common scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898687#comment-13898687 ] Hudson commented on YARN-1498: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5150/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Common scheduler changes for moving apps between queues --- Key: YARN-1498 URL: https://issues.apache.org/jira/browse/YARN-1498 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1498-1.patch, YARN-1498-addendum.patch, YARN-1498.patch, YARN-1498.patch This JIRA is to track changes that aren't in particular schedulers but that help them support moving apps between queues. In particular, it makes sure that QueueMetrics are properly updated when an app changes queue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1499) Fair Scheduler changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898683#comment-13898683 ] Hudson commented on YARN-1499: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5150/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Fair Scheduler changes for moving apps between queues - Key: YARN-1499 URL: https://issues.apache.org/jira/browse/YARN-1499 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1499-1.patch, YARN-1499-2.patch, YARN-1499.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1496) Protocol additions to allow moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898685#comment-13898685 ] Hudson commented on YARN-1496: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5150/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Protocol additions to allow moving apps between queues -- Key: YARN-1496 URL: https://issues.apache.org/jira/browse/YARN-1496 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1496-1.patch, YARN-1496-2.patch, YARN-1496-3.patch, YARN-1496-4.patch, YARN-1496-5.patch, YARN-1496-6.patch, YARN-1496.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1497) Expose moving apps between queues on the command line
[ https://issues.apache.org/jira/browse/YARN-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898686#comment-13898686 ] Hudson commented on YARN-1497: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5150/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Expose moving apps between queues on the command line - Key: YARN-1497 URL: https://issues.apache.org/jira/browse/YARN-1497 Project: Hadoop YARN Issue Type: Sub-task Components: client Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1497-1.patch, YARN-1497.patch, YARN-1497.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1504) RM changes for moving apps between queues
[ https://issues.apache.org/jira/browse/YARN-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898684#comment-13898684 ] Hudson commented on YARN-1504: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5150 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5150/]) Move YARN-1496, YARN-1498, YARN-1504, YARN-1499, and YARN-1497 under 2.4 in CHANGES.txt (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567487) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt RM changes for moving apps between queues - Key: YARN-1504 URL: https://issues.apache.org/jira/browse/YARN-1504 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 3.0.0 Attachments: YARN-1504-1.patch, YARN-1504.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1717) Misc improvements to leveldb timeline store
[ https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898703#comment-13898703 ] Hadoop QA commented on YARN-1717: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628406/YARN-1717.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3079//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3079//console This message is automatically generated. Misc improvements to leveldb timeline store --- Key: YARN-1717 URL: https://issues.apache.org/jira/browse/YARN-1717 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Attachments: YARN-1717.1.patch, YARN-1717.2.patch The leveldb timeline store implementation needs the following: * better documentation of its internal structures * braces for all control flow statements * simple locking to prevent issues related to concurrent writes * internal changes to enable deleting entities ** never overwrite existing primary filter entries ** add hidden reverse pointers to related entities -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1719) ATSWebServices produces jersey warnings
[ https://issues.apache.org/jira/browse/YARN-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898716#comment-13898716 ] Hudson commented on YARN-1719: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5151 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5151/]) YARN-1719. Fixed the root path related Jersey warnings produced in ATSWebServices. Contributed by Billie Rinaldi. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1567498) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/ATSWebServices.java ATSWebServices produces jersey warnings --- Key: YARN-1719 URL: https://issues.apache.org/jira/browse/YARN-1719 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Fix For: 2.4.0 Attachments: YARN-1719.1.patch These don't appear to affect how the web services work, but the following warnings are logged: {noformat} WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ letRequest,javax.servlet.http.HttpServletResponse), with URI template, /, is t reated as a resource method WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE ntities), with URI template, /, is treated as a resource method {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[ https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1553: -- Target Version/s: 2.4.0 Do not use HttpConfig.isSecure() in YARN Key: YARN-1553 URL: https://issues.apache.org/jira/browse/YARN-1553 Project: Hadoop YARN Issue Type: Bug Reporter: Haohui Mai Assignee: Haohui Mai Attachments: YARN-1553.000.patch, YARN-1553.001.patch, YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch, YARN-1553.005.patch, YARN-1553.006.patch HDFS-5305 and related jira decide that each individual project will have their own configuration on http policy. {{HttpConfig.isSecure}} is a global static method which does not fit the design anymore. The same functionality should be moved into the YARN code base. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1553) Do not use HttpConfig.isSecure() in YARN
[ https://issues.apache.org/jira/browse/YARN-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898774#comment-13898774 ] Hadoop QA commented on YARN-1553: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628392/YARN-1553.006.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 10 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy: org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.yarn.client.TestRMFailover The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy: org.apache.hadoop.yarn.client.api.impl.TestNMClient {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3078//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3078//console This message is automatically generated. Do not use HttpConfig.isSecure() in YARN Key: YARN-1553 URL: https://issues.apache.org/jira/browse/YARN-1553 Project: Hadoop YARN Issue Type: Bug Reporter: Haohui Mai Assignee: Haohui Mai Attachments: YARN-1553.000.patch, YARN-1553.001.patch, YARN-1553.002.patch, YARN-1553.003.patch, YARN-1553.004.patch, YARN-1553.005.patch, YARN-1553.006.patch HDFS-5305 and related jira decide that each individual project will have their own configuration on http policy. {{HttpConfig.isSecure}} is a global static method which does not fit the design anymore. The same functionality should be moved into the YARN code base. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-1641) ZK store should attempt a write periodically to ensure it is still Active
[ https://issues.apache.org/jira/browse/YARN-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1641: --- Attachment: yarn-1641-2.patch Added thread name. ZK store should attempt a write periodically to ensure it is still Active - Key: YARN-1641 URL: https://issues.apache.org/jira/browse/YARN-1641 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.3.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: yarn-1641-1.patch, yarn-1641-2.patch Fencing in ZK store kicks in when the RM tries to write something to the store. If the RM doesn't write anything to the store, it doesn't get fenced and can continue to assume being the Active. By periodically writing a file (say, every RM_ZK_TIMEOUT_MS seconds), we can ensure it gets fenced. -- This message was sent by Atlassian JIRA (v6.1.5#6160)