[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13895517#comment-13895517 ] Hudson commented on YARN-1493: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #475 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/475/]) YARN-1493,YARN-1490,YARN-1041, YARN-1166,YARN-1566,YARN-1689,YARN-1661 are reverted from branch-2.3. Updating YARN's CHANGES.txt. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1565805) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.4.0 Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch, YARN-1493.8.patch, YARN-1493.9.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13895553#comment-13895553 ] Hudson commented on YARN-1493: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1667 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1667/]) YARN-1493,YARN-1490,YARN-1041, YARN-1166,YARN-1566,YARN-1689,YARN-1661 are reverted from branch-2.3. Updating YARN's CHANGES.txt. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1565805) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.4.0 Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch, YARN-1493.8.patch, YARN-1493.9.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13895588#comment-13895588 ] Hudson commented on YARN-1493: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1692 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1692/]) YARN-1493,YARN-1490,YARN-1041, YARN-1166,YARN-1566,YARN-1689,YARN-1661 are reverted from branch-2.3. Updating YARN's CHANGES.txt. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1565805) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.4.0 Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch, YARN-1493.8.patch, YARN-1493.9.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13895065#comment-13895065 ] Hudson commented on YARN-1493: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5130 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5130/]) YARN-1493,YARN-1490,YARN-1041, YARN-1166,YARN-1566,YARN-1689,YARN-1661 are reverted from branch-2.3. Updating YARN's CHANGES.txt. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1565805) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.4.0 Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch, YARN-1493.8.patch, YARN-1493.9.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861428#comment-13861428 ] Hudson commented on YARN-1493: -- FAILURE: Integrated in Hadoop-Yarn-trunk #441 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/441/]) YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize app-attempts separately from apps. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554896) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptRejectedEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ActiveUsersManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAttemptAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppRemovedSchedulerEvent.java *
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861513#comment-13861513 ] Hudson commented on YARN-1493: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1633 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1633/]) YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize app-attempts separately from apps. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554896) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptRejectedEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ActiveUsersManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAttemptAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppRemovedSchedulerEvent.java *
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861558#comment-13861558 ] Hudson commented on YARN-1493: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1658 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1658/]) YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize app-attempts separately from apps. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554896) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptRejectedEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ActiveUsersManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAttemptAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppRemovedSchedulerEvent.java *
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860738#comment-13860738 ] Hudson commented on YARN-1493: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4951 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4951/]) YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize app-attempts separately from apps. Contributed by Jian He. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1554896) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptRejectedEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ActiveUsersManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAttemptAddedSchedulerEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppRemovedSchedulerEvent.java *
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13858460#comment-13858460
]
Hadoop QA commented on YARN-1493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620786/YARN-1493.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2754//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch,
YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch,
YARN-1493.8.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13858480#comment-13858480
]
Hadoop QA commented on YARN-1493:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620787/YARN-1493.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2755//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2755//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch,
YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch,
YARN-1493.8.patch, YARN-1493.9.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857672#comment-13857672 ] Jian He commented on YARN-1493: --- New patch changes: - Make schedulers send the App_accepted/App_rejected event to the RMApp instead of RMAppAttempt. - Create two new events AppAddedSchedulerEvent and AppRemovedSchedulerEvent for adding and removing apps in the schedulers. - Change the state transition to start a new attempt until the app is accepted by the scheduler. - Rename SchedulerApplication to SchedulerApplicationAttempt, and created a new SchedulerApplication for keeping track of the app-specific info. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857677#comment-13857677
]
Hadoop QA commented on YARN-1493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620638/YARN-1493.7.patch
against trunk revision .
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2741//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch,
YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857702#comment-13857702 ] Sandy Ryza commented on YARN-1493: -- Thanks for clarifying about the FinalTransition, Jian. Took another look at the patch and I'm +1 once it applies and the findbugs issues are resolved. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857831#comment-13857831 ] Jian He commented on YARN-1493: --- bq. When submission is rejected by a parent queue, you need to call removeApplication. This existed before but your patch removed it. The earlier addApplication is renamed to addApplicationAttempt, this addApplicationAttempt is called when the SchedulerAttemptAddedEvent comes. So we are not adding or removing any application data structure in the leaf queue at all, we are adding/removing attempt in the leaf queue. bq. finishApplicationAttempt: Should Inform the parent queue so that it can call finishApplicationAttempt itself. Similarly for submitApplicationAttempt. ParetQueue’s finishApplicationAttempt and submitApplicationAttempt logic is empty, ParetQueue only deal with app-specific logic in the current implementation. Do we still want to call parentQueue in attempt-specific APIs? bq. We shouldn’t move to ACCEPTED directly before informing scheduler in case of recovery? YARN-1507 is saving the application after app is accepted. So after YARN-1507, an app is saved meaning it is accepted. Maybe leave it for now and fix it in YARN-1507 ? bq. RMAppEventType.ATTEMPT_FAILED event should not come in at ACCEPTED state? This is possible because, RMAppRecoveredTransition is changed to return ACCEPTED state, and waiting for the AttemptFailed event to come (waiting for the previous AM to exit) I changed it to ACCEPTED state instead of RUNNING because as said after YARN-1507, an app is saved meaning it is ACCEPTED. the app may not necessarily be at RUNNING state earlier. bq. When can this happen? During recovery? May be we should fix that correctly? This can happen because I changed app to return ACCEPTED state on recovery, and on recovery the app once again go through the scheduler and triggers one more APP_ACCEPTED event at ACCEPTED state. bq. TestFairScheduler: Why the conditional? because testAclSubmitApplication is testing app2 to be null (AssertNull(The application was allowed, app2)), the app is rejected and no app exists. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857839#comment-13857839 ] Jian He commented on YARN-1493: --- bq. The information about submission to various queues is lost? It is not needed? Investigated that the queue passed in the earlier code is only used in ParentQueue.submitApplication for comparing the queue's name with the parentQueue's name for validation. None of the test is using this requirement. And Also, for every single test, the test is using the same queue. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857888#comment-13857888 ] Vinod Kumar Vavilapalli commented on YARN-1493: --- bq. The earlier addApplication is renamed to addApplicationAttempt, this addApplicationAttempt is called when the SchedulerAttemptAddedEvent comes. So we are not adding or removing any application data structure in the leaf queue at all, we are adding/removing attempt in the leaf queue. Yeah, that is the current state. Let's do the same to future-proof it. bq. ParetQueue’s finishApplicationAttempt and submitApplicationAttempt logic is empty, ParetQueue only deal with app-specific logic in the current implementation. Do we still want to call parentQueue in attempt-specific APIs? Again, that's today. Let's do it the way one would see it in future. bq. YARN-1507 is saving the application after app is accepted. So after YARN-1507, an app is saved meaning it is accepted. Maybe leave it for now and fix it in YARN-1507 ? Sure, you should leave a code-comment nonetheless pointing to the JIRA. bq. I changed it to ACCEPTED state instead of RUNNING because as said after YARN-1507, an app is saved meaning it is ACCEPTED. the app may not necessarily be at RUNNING state earlier. Again, let's leave a code comment saying the same. bq. This can happen because I changed app to return ACCEPTED state on recovery, and on recovery the app once again go through the scheduler and triggers one more APP_ACCEPTED event at ACCEPTED state. Ditto. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857904#comment-13857904 ] Vinod Kumar Vavilapalli commented on YARN-1493: --- bq. When submission is rejected by a parent queue, you need to call removeApplication. This existed before but your patch removed it. I withdraw this comment, misread it. Also, in LeafQueue, ParentQueue etc, please put @Override annotations where necessary. It avoids confusion about internal and implemented methods. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch, YARN-1493.7.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856070#comment-13856070
]
Hadoop QA commented on YARN-1493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620290/YARN-1493.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 9 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2719//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2719//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2719//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch,
YARN-1493.4.patch, YARN-1493.5.patch, YARN-1493.6.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855379#comment-13855379 ] Jian He commented on YARN-1493: --- bq. I don't see an AppRemovedSchedulerEvent fired during an AppKilledTransition. AppKilledTransition is calling FinalTransition which fires AppRemovedSchedulerEvent event. bq. Still a few places where second line only has two instead of four spaces of indentation. looks like some problem with my formatter, will fix it. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855407#comment-13855407
]
Hadoop QA commented on YARN-1493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620128/YARN-1493.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 12 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2716//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2716//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2716//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch,
YARN-1493.4.patch, YARN-1493.5.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855418#comment-13855418 ] Jian He commented on YARN-1493: --- The new patch included the patch for YARN-1490, as while working on them, they seem tightly coupled to me. will separate it if necessary. What this patch has done is to share the containers across AMs, basically by copying needed objects from first attempt to the second attempt when the attempt is failing over. Summarize the patch: - Make schedulers send the App_accepted/App_rejected event to the RMApp instead of RMAppAttempt. - Create two new events AppAddedSchedulerEvent and AppRemovedSchedulerEvent for adding and removing apps in the schedulers. - Change the state transition to start a new attempt until the app is accepted by the scheduler. - Create a new SchedulerApplication and rename the current SchedulerApplication to SchedulerApplicationAttempt. - Create a field in AppSubmissionContext to indicate whether to clean the containers on AM failure or not. - Copy the data structures inside SchedulerApplicationAttempt over in the case that new attempt is recovering the failed attempt’s scheduler info. - Similarly, copy the needed data structures inside RMAppAttempt over in the case that new attempt is recovering the failed RMAppAttempt info. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch, YARN-1493.5.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13852499#comment-13852499 ] Vinod Kumar Vavilapalli commented on YARN-1493: --- Tx for working on this! This is a big improvement and a huge stride forward for many more things to come. Started looking at the patch. Tx also for restricting the changes. There's a lot more we can do, more refactoring, moving more stuff into app vs appattempt, more renames etc. But let's continue to keep those to follow up JIRAs for helping reviews. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850227#comment-13850227
]
Hadoop QA commented on YARN-1493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619046/YARN-1493.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2675//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2675//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850881#comment-13850881
]
Hadoop QA commented on YARN-1493:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619152/YARN-1493.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2679//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2679//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850929#comment-13850929
]
Sandy Ryza commented on YARN-1493:
--
Thanks Jian. A question and some stylistic issues. Other than that, after
SchedulerApplicationWrapper is replaced with SchedulerApplication, the patch
looks good to me.
Will the scheduler ever be in a state where an application has been removed but
an attempt from that app still remains? My understanding is no, but I wanted
to make sure.
{code}
+public class SchedulerApplicationWrapper {
+
+ Queue queue;
+ String user;
+
{code}
These member variables should be private and final (or labeled
@VisibleForTesting if that's why they're not).
{code}
+ this.rmContext.getDispatcher().getEventHandler()
+.handle(new RMAppRejectedEvent(applicationId, message));
{code}
Second line should be indented four spaces. This applies in a number of other
places as well.
{code}
+ protected MapApplicationId, SchedulerApplicationWrapper applications = new
+ ConcurrentHashMapApplicationId, SchedulerApplicationWrapper();
{code}
new should be on the second line
{code}
- /**
- * Add a new application to the scheduler, with a given id, queue name, and
- * user. This will accept a new app even if the user or queue is above
- * configured limits, but the app will not be marked as runnable.
- */
{code}
No reason to remove these comments for FairScheduler.addApplication.
addApplicationAttempt should have updated comments.
{code}
+LOG.info(Succeeded to submit application + applicationId + to queue
++ queueName + from user + user);
{code}
Should be Accepting application applicationId from user userName in queue
queueName
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851051#comment-13851051 ] Jian He commented on YARN-1493: --- Fixed recent comments, and rename SchedulerApplication to SchedulerApplicationAttempt , and created a new SchedulerApplication for keeping track of the app-specific info. bq. Will the scheduler ever be in a state where an application has been removed but an attempt from that app still remains? I also think this should not happen. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch, YARN-1493.4.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851069#comment-13851069
]
Sandy Ryza commented on YARN-1493:
--
I took another look at the patch and noticed something else - is the app
removed from the scheduler in all cases when the . I.e. I don't see an
AppRemovedSchedulerEvent fired during an AppKilledTransition.
{code}
+ app.handler.handle(new AppRemovedSchedulerEvent(app.applicationId, app
+.getState()));
{code}
Still a few places where second line only has two instead of four spaces of
indentation.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch,
YARN-1493.4.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851088#comment-13851088
]
Hadoop QA commented on YARN-1493:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619186/YARN-1493.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2685//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2685//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch, YARN-1493.2.patch, YARN-1493.3.patch,
YARN-1493.4.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849638#comment-13849638 ] Sandy Ryza commented on YARN-1493: -- Before this, could the scheduler ever be running two attempts from the same application at the same time? Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849653#comment-13849653
]
Hadoop QA commented on YARN-1493:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618951/YARN-1493.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2667//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-sls.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2667//console
This message is automatically generated.
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849676#comment-13849676 ] Jian He commented on YARN-1493: --- bq. Before this, could the scheduler ever be running two attempts from the same application at the same time? we supposedly should not have two attempts for the same application running, YARN-1210 was fixing this. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[
https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849761#comment-13849761
]
Sandy Ryza commented on YARN-1493:
--
Thanks for clarifying Jian.
bq. will rename this to SchedulerApplication, and rename the current
SchedulerApplication to SchedulerApplicationAttempt
Do you plan to do that in this patch? If not, why not? Right now the class
isn't actually wrapping anything so we should at least change its name.
Didn't do a full review, but a couple things I noticed:
{code}
+SchedulerApplicationWrapper application =
+applications.get(applicationAttemptId.getApplicationId());
+String user =
+applications.get(applicationAttemptId.getApplicationId()).getUser();
{code}
Should use application to get the user.
{code}
ApplicationId appId = BuilderUtils.newApplicationId(100, 1);
ApplicationAttemptId appAttemptId = BuilderUtils.newApplicationAttemptId(
appId, 1);
-SchedulerEvent event =
-new AppAttemptAddedSchedulerEvent(appAttemptId, default, user);
-fs.handle(event);
+SchedulerEvent appEvent =
+new AppAddedSchedulerEvent(appAttemptId.getApplicationId(), default,
+ user);
{code}
Can use appId instead of appAttemptId.getApplicationId()
Schedulers don't recognize apps separately from app-attempts
Key: YARN-1493
URL: https://issues.apache.org/jira/browse/YARN-1493
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-1493.1.patch
Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new
app-level events to the scheduler and separate the app-level logic out. This
is good for work-preserving AM restart, RM restart, and also needed for
differentiating app-level metrics and attempt-level metrics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849768#comment-13849768 ] Jian He commented on YARN-1493: --- bq. Do you plan to do that in this patch? If not, why not? Forgot to say I just want to keep the current patch small and easy for initial review. will fix it in this jira. Thanks for the comments, will address it. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1493) Schedulers don't recognize apps separately from app-attempts
[ https://issues.apache.org/jira/browse/YARN-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849824#comment-13849824 ] Vinod Kumar Vavilapalli commented on YARN-1493: --- From what I can see, this patch is clearly conflicting with YARN-1446. Linking tickets. Schedulers don't recognize apps separately from app-attempts Key: YARN-1493 URL: https://issues.apache.org/jira/browse/YARN-1493 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-1493.1.patch Today, scheduler is tied to attempt only. We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
