[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745784#comment-14745784
]
Hudson commented on YARN-3635:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #394 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/394/])
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe:
rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745779#comment-14745779
]
Hudson commented on YARN-3635:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2336 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2336/])
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe:
rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
* hadoop-yarn-project/CHANGES.txt
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745405#comment-14745405
]
Hudson commented on YARN-3635:
--
ABORTED: Integrated in Hadoop-Mapreduce-trunk-Java8 #388 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/388/])
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe:
rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745406#comment-14745406
]
Hudson commented on YARN-3635:
--
ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #372 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/372/])
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe:
rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745398#comment-14745398
]
Hudson commented on YARN-3635:
--
ABORTED: Integrated in Hadoop-Yarn-trunk #1127 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1127/])
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe:
rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745376#comment-14745376
]
Hudson commented on YARN-3635:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2312 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2312/])
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe:
rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745265#comment-14745265
]
Hudson commented on YARN-3635:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8453 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8453/])
YARN-3635. Refactored current queue mapping implementation in CapacityScheduler
to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe:
rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745249#comment-14745249
]
Jian He commented on YARN-3635:
---
Committed to trunk and branch-2, thanks Wangda !
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743622#comment-14743622
]
Jian He commented on YARN-3635:
---
+1, committing soon if no further comments.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743209#comment-14743209
]
Jian He commented on YARN-3635:
---
lgtm
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743087#comment-14743087
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 48s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 1s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 12s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 50s | The applied patch generated
14 new checkstyle issues (total was 234, now 241). |
| {color:red}-1{color} | whitespace | 0m 6s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 54m 31s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 94m 30s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755647/YARN-3635.8.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e1b1d7e |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9118/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14742925#comment-14742925
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 16s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 48s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 46s | The applied patch generated
14 new checkstyle issues (total was 234, now 241). |
| {color:red}-1{color} | whitespace | 0m 6s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 45m 40s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 84m 5s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
| | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens |
| |
hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions |
| | hadoop.yarn.server.resourcemanager.TestApplicationCleanup |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher |
| | hadoop.yarn.server.resourcemanager.TestRM |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| | hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes |
| | hadoop.yarn.server.resourcemanager.TestResourceManager |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| | hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755647/YARN-3635.8.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 332b520 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9112/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9112/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch,
> YARN-3635.8.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after sub
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735471#comment-14735471
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 8s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 52s | The applied patch generated
14 new checkstyle issues (total was 236, now 242). |
| {color:red}-1{color} | whitespace | 0m 3s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 54m 13s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 93m 49s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12754296/YARN-3635.7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 970daaa |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9040/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9040/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch, YARN-3635.7.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715222#comment-14715222
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745158/YARN-3635.6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a4d9acc |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8918/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703783#comment-14703783
]
Wangda Tan commented on YARN-3635:
--
[~sandyr], any comments on reply above?
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630139#comment-14630139
]
Wangda Tan commented on YARN-3635:
--
Hi [~sandyr],
Thanks for your comments, actually I have read
QueuePlacementPolicy/QueuePlacementRule from FS before working on this patch.
The generic design fo this patch is based on FS's queue placement policy
structure, but also with some changes.
To your comments:
bq. Is a common way of configuration proposed?
No common configuration, it only defined a set of common interfaces. Since
FS/CS have very different ways to configuration, now rules are created by
different schedulers, see CapacityScheduler#updatePlacementRules as an example.
bq. What steps are required for the Fair Scheduler to integrate with this?
1) Port existing rules to new APIs defined in the patch, this should be simple
2) Change configuration implementation to instance new defined PlacementRule,
you may not need to change existing configuration items itself.
3) Change FS workflow, with this patch, queue mapping is happened before submit
to scheduler. Remove queue mapping related logics from FS and create queue if
needed.
bq. Each placement rule gets the chance to assign the app to a queue, reject
the app, or pass. If it passes, the next rule gets a chance.
New APIs are very similar:
Non-null is determined
Null is not determined
Throw exception when rejected.
You can take a look at
{{org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementRule}}
bq. A placement rule can base its decision on:
bq.
Yes you can do all of them with the new API except "The set of queues given in
the Fair Scheduler configuration.":
I was thinking necessarity of passing set of queues in the interface. In
existing implementations of QueuePlacementPolicy, FS queues are only used to
check mapped queue's existence. I would prefer to delay the check to submit to
scheduler. See my next comment about "create" flag for more details.
Another reason of not passing queue names set via interface is, queues are very
dynamic. For example, if user wants to submit application to queue with lowest
utilization, queue names set may not enough. I would prefer to let rule choose
to get what need from scheduler.
bq. Rules are marked as "terminal" if they will never pass. This helps to avoid
misconfigurations where users place rules after terminal rules.
I'm not sure if is it necessary. I think terminal or not should be determined
by runtime, but I'm OK if you think it's must to have.
bq. Rules have a "create" attribute which determines whether they can create a
new queue or whether they must assign to existing queues.
I think queue is create-able or not should be determined by scheduler, it
should be a part of scheduler configuration instead of rule itself.
You can put "create" to your implemented rules without any issue, but I prefer
not to expose it to public interface.
bq. Currently the set of placement rules is limited to what's implemented in
YARN. I.e. there's no public pluggable rule support.
Agree, this is one thing we need to do in the future. For now, we can make
queue mapping happens in a central place first.
bq. Are there places where my summary would not describe what's going on in
this patch?
I think it should covers most of my patch, you can also take a look at my patch
to see if anything unexpected :).
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629141#comment-14629141
]
Sandy Ryza commented on YARN-3635:
--
BTW I got all this from QueuePlacementPolicy and QueuePlacementRule, which are
pretty quick reads if you want to take a look.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629139#comment-14629139
]
Sandy Ryza commented on YARN-3635:
--
[~leftnoteasy], apologies for this quick drive-by review - I am currently
traveling.
The JIRA appears to be lacking a design-doc and I wasn't able to find
documentation in the patch. The patch should ultimately include some detailed
documentation, but I don't want to ask this of you before OKing the approach.
In light of this, a few questions:
* What steps are required for the Fair Scheduler to integrate with this?
* Is a common way of configuration proposed?
* How does this differ from the current Fair Scheduler model? To summarize:
** The FS model consists of a sequence of placement rules that the app is
passed through.
** Each placement rule gets the chance to assign the app to a queue, reject the
app, or pass. If it passes, the next rule gets a chance.
** A placement rule can base its decision on:
*** The submitting user.
*** The set of groups the submitting user belongs to.
*** The queue requested in the app submission.
*** A set of configuration options that are specific to the rule.
*** The set of queues given in the Fair Scheduler configuration.
** Rules are marked as "terminal" if they will never pass. This helps to avoid
misconfigurations where users place rules after terminal rules.
** Rules have a "create" attribute which determines whether they can create a
new queue or whether they must assign to existing queues.
** Currently the set of placement rules is limited to what's implemented in
YARN. I.e. there's no public pluggable rule support.
I noticed from Vinod's comment that this patch follows a similar structure.
Are there places where my summary would not describe what's going on in this
patch?
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Tan, Wangda
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627086#comment-14627086
]
Wangda Tan commented on YARN-3635:
--
[~kasha]/[~sandyr], do you want to take a look at latest patch?
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627036#comment-14627036
]
Jian He commented on YARN-3635:
---
newest patch looks good to me, + !
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625732#comment-14625732
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 15s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 49s | The applied patch generated
14 new checkstyle issues (total was 234, now 241). |
| {color:red}-1{color} | whitespace | 0m 4s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 51m 7s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 23s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745158/YARN-3635.6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a431ed9 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8526/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8526/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch, YARN-3635.6.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625621#comment-14625621
]
Jian He commented on YARN-3635:
---
lgtm overall,
- the queuePlacementManager should be added in RMActiveServicesContext which
includes all active services.
{code}
private PlacementManager queuePlacementManager;
{code}
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625620#comment-14625620
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 10s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 52s | The applied patch generated
18 new checkstyle issues (total was 210, now 221). |
| {color:red}-1{color} | whitespace | 0m 3s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 32s | The patch appears to introduce 3
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 51m 31s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 40s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-resourcemanager |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12741145/YARN-3635.5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9ef03a4 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8525/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8525/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596894#comment-14596894
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 46s | The applied patch generated
18 new checkstyle issues (total was 204, now 215). |
| {color:red}-1{color} | whitespace | 0m 3s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 30s | The patch appears to introduce 3
new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 50m 14s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 34s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12741145/YARN-3635.5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 077250d |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8316/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8316/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch, YARN-3635.5.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596620#comment-14596620
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 14s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 48s | The applied patch generated 2
additional warning messages. |
| {color:red}-1{color} | release audit | 0m 18s | The applied patch generated
4 release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 47s | The applied patch generated
18 new checkstyle issues (total was 204, now 215). |
| {color:red}-1{color} | whitespace | 0m 3s | The patch has 15 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 27s | The patch appears to introduce 3
new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 61m 8s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 99m 39s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12741096/YARN-3635.4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 445b132 |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/diffJavadocWarnings.txt
|
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8311/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch,
> YARN-3635.4.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546108#comment-14546108
]
Vinod Kumar Vavilapalli commented on YARN-3635:
---
Okay, I didn't think you were really meaning the JIRA title literally.
In general, I think we should have a separate component sitting next to
ClientService responsible for mapping apps to queues. I've seen lot more
use-cases beyond what we have today in either of the schedulers - it will be
useful to start laying this out with an ultimate intention to potentially make
it an end-user visible abstraction.
Here's what I propose
- PlacementRule represents a rule with the following interface:
{{PlacementAction matches(Application)}}. Examples includes matching queue-name
argument, user-name to queue mapping etc.
- PlacementAction represents the action to take: simplest is to submit to a
queue.
- PlacementRuleChain: A chain of PlacementRules
- PlacementRuleChain getting initialized from scheduler specific config files
for now
- A PlacementManager which keeps track of PlacementRuleChain and takes action
on incoming apps
Thoughts?
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545909#comment-14545909
]
Wangda Tan commented on YARN-3635:
--
[~kasha],
Actually no, this patch doesn't do anything to RM when using FS, RMAppManager
use the original user specified queue name to do initialization, and FS can
continue applying QueuePlacementPolicy after app added to scheduler. FS tests
can all get passed also.
Any other thoughts?
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544757#comment-14544757
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 32s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 48s | The applied patch generated 1
new checkstyle issues (total was 210, now 211). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 8 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 18s | The patch appears to introduce 1
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 50m 2s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 86m 11s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| | Inconsistent synchronization of
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.isHDFS;
locked 66% of time Unsynchronized access at FileSystemRMStateStore.java:66%
of time Unsynchronized access at FileSystemRMStateStore.java:[line 156] |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12733003/YARN-3635.3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9a2a955 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7945/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7945/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544703#comment-14544703
]
Karthik Kambatla commented on YARN-3635:
With FairScheduler, if this patch is applied, does the QueuePlacementPolicy
cease to work?
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch, YARN-3635.3.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544533#comment-14544533
]
Jian He commented on YARN-3635:
---
looks good overall, getQueueNameOfNewApp, may be call it getMappedQueue ?
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544037#comment-14544037
]
Wangda Tan commented on YARN-3635:
--
Failed test is not related to this change, which is tracked by YARN-3504
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543073#comment-14543073
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 45s | The applied patch generated 1
new checkstyle issues (total was 210, now 211). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 8 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | yarn tests | 49m 51s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 86m 12s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732726/YARN-3635.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e85044 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7929/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7929/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch, YARN-3635.2.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542568#comment-14542568
]
Hadoop QA commented on YARN-3635:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 7s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 9m 9s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 9s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 55s | The applied patch generated 1
new checkstyle issues (total was 210, now 211). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 8 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 46s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 57s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 53s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | yarn tests | 46m 29s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.server.resourcemanager.TestClientRMService |
| Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732630/YARN-3635.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 065d8f2 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7923/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7923/console |
This message was automatically generated.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3635.1.patch
>
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540930#comment-14540930
]
Wangda Tan commented on YARN-3635:
--
[~kasha], Thanks for supporting this. Sure I will ping you once I get a patch.
[~vinodkv], I will take a look at implementation in different schedulers, I
will focus on interface itself in the patch first, and to see we can address
the common logic merging together in the patch.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540586#comment-14540586
]
Vinod Kumar Vavilapalli commented on YARN-3635:
---
+10 for this.
In addition to the interfaces themselves, I'd like us to consolidate the
concrete mapping-rules that we have in each scheduler. Ideally, we only need
one set of rules acceptable by all schedulers. If not that, I'd live with ~80%
common rules.
BTW, thematically this fits into YARN-1317, making it a sub-task.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540393#comment-14540393
]
Karthik Kambatla commented on YARN-3635:
I am in favor of doing this, but would like to be extra careful. Can we make
sure either [~sandyr] or I get to review this before it is committed.
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3635) Get-queue-mapping should be a common interface of YarnScheduler
[
https://issues.apache.org/jira/browse/YARN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540371#comment-14540371
]
Wangda Tan commented on YARN-3635:
--
+[~vinodkv], [~kasha], [~jlowe], [~jianhe]
> Get-queue-mapping should be a common interface of YarnScheduler
> ---
>
> Key: YARN-3635
> URL: https://issues.apache.org/jira/browse/YARN-3635
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>
> Currently, both of fair/capacity scheduler support queue mapping, which makes
> scheduler can change queue of an application after submitted to scheduler.
> One issue of doing this in specific scheduler is: If the queue after mapping
> has different maximum_allocation/default-node-label-expression of the
> original queue, {{validateAndCreateResourceRequest}} in RMAppManager checks
> the wrong queue.
> I propose to make the queue mapping as a common interface of scheduler, and
> RMAppManager set the queue after mapping before doing validations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
