[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2920: -- Labels: 2.6.1-candidate (was: ) Fix Version/s: 2.6.1 Pulled this into 2.6.1 as a dependency. Had to fix some import conflicts and merge conflicts. Ran all ResourceManager tests before pushing this in. > CapacityScheduler should be notified when labels on nodes changed > - > > Key: YARN-2920 > URL: https://issues.apache.org/jira/browse/YARN-2920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-2920.1.patch, YARN-2920.2.patch, YARN-2920.3.patch, > YARN-2920.4.patch, YARN-2920.5.patch, YARN-2920.6.patch > > > Currently, labels on nodes changes will only be handled by > RMNodeLabelsManager, but that is not enough upon labels on nodes changes: > - Scheduler should be able to do take actions to running containers. (Like > kill/preempt/do-nothing) > - Used / available capacity in scheduler should be updated for future > planning. > We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2920: -- Attachment: YARN-2920-branch-2.6.1.txt Attaching patch that I committed to 2.6.1. > CapacityScheduler should be notified when labels on nodes changed > - > > Key: YARN-2920 > URL: https://issues.apache.org/jira/browse/YARN-2920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-2920-branch-2.6.1.txt, YARN-2920.1.patch, > YARN-2920.2.patch, YARN-2920.3.patch, YARN-2920.4.patch, YARN-2920.5.patch, > YARN-2920.6.patch > > > Currently, labels on nodes changes will only be handled by > RMNodeLabelsManager, but that is not enough upon labels on nodes changes: > - Scheduler should be able to do take actions to running containers. (Like > kill/preempt/do-nothing) > - Used / available capacity in scheduler should be updated for future > planning. > We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[
https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-2920:
-
Attachment: YARN-2920.4.patch
[~jianhe],
Thanks for your comments, I've updated patch addressed all your suggestions,
for your comment:
bq. how about containers running on a node without label, and now we are adding
a label.
Now we will also kill containers on that node, this will be changed after we
get YARN-2498 in. I added a TODO note at
{{CapacityScheduler.updateLabelsOnNode}}.
Please kindly review,
Wangda
CapacityScheduler should be notified when labels on nodes changed
-
Key: YARN-2920
URL: https://issues.apache.org/jira/browse/YARN-2920
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2920.1.patch, YARN-2920.2.patch, YARN-2920.3.patch,
YARN-2920.4.patch
Currently, labels on nodes changes will only be handled by
RMNodeLabelsManager, but that is not enough upon labels on nodes changes:
- Scheduler should be able to do take actions to running containers. (Like
kill/preempt/do-nothing)
- Used / available capacity in scheduler should be updated for future
planning.
We need add a new event to pass such updates to scheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2920: - Attachment: YARN-2920.5.patch bq. getUsedResources- getUsedResourcesByLabel And bq. RMNodeLabelsManager constructor can pass rmContext, instead of a separate setRMDispatcher method Make sense to me, updated. bq. AM container is killed as well, should we not kill the am container until the max-am-percentage is met, similar to preemption? This needs update internal used resource for LeafQueue/ParentQueue. With YARN-2498, containers will not be immediately killed, and preemption policy will handle that, AM is already last killed by preemption policy. Thanks, Wangda CapacityScheduler should be notified when labels on nodes changed - Key: YARN-2920 URL: https://issues.apache.org/jira/browse/YARN-2920 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2920.1.patch, YARN-2920.2.patch, YARN-2920.3.patch, YARN-2920.4.patch, YARN-2920.5.patch Currently, labels on nodes changes will only be handled by RMNodeLabelsManager, but that is not enough upon labels on nodes changes: - Scheduler should be able to do take actions to running containers. (Like kill/preempt/do-nothing) - Used / available capacity in scheduler should be updated for future planning. We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2920: - Attachment: YARN-2920.6.patch Fixed test failure (TestResourceTracker is related, but TestAbstractYarnScheduler cannot reproduce locally). Findbugs warning not related. CapacityScheduler should be notified when labels on nodes changed - Key: YARN-2920 URL: https://issues.apache.org/jira/browse/YARN-2920 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2920.1.patch, YARN-2920.2.patch, YARN-2920.3.patch, YARN-2920.4.patch, YARN-2920.5.patch, YARN-2920.6.patch Currently, labels on nodes changes will only be handled by RMNodeLabelsManager, but that is not enough upon labels on nodes changes: - Scheduler should be able to do take actions to running containers. (Like kill/preempt/do-nothing) - Used / available capacity in scheduler should be updated for future planning. We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2920: - Attachment: YARN-2920.3.patch Updated patch, fixed release audit warnings and javac warnings. Test failure is not related. CapacityScheduler should be notified when labels on nodes changed - Key: YARN-2920 URL: https://issues.apache.org/jira/browse/YARN-2920 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2920.1.patch, YARN-2920.2.patch, YARN-2920.3.patch Currently, labels on nodes changes will only be handled by RMNodeLabelsManager, but that is not enough upon labels on nodes changes: - Scheduler should be able to do take actions to running containers. (Like kill/preempt/do-nothing) - Used / available capacity in scheduler should be updated for future planning. We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2920: - Attachment: YARN-2920.2.patch Updated patch, dropped some unnecessary refactoring code which will cause deadlock (tracked by YARN-2925). Resolved UT failures. CapacityScheduler should be notified when labels on nodes changed - Key: YARN-2920 URL: https://issues.apache.org/jira/browse/YARN-2920 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2920.1.patch, YARN-2920.2.patch Currently, labels on nodes changes will only be handled by RMNodeLabelsManager, but that is not enough upon labels on nodes changes: - Scheduler should be able to do take actions to running containers. (Like kill/preempt/do-nothing) - Used / available capacity in scheduler should be updated for future planning. We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2920: - Issue Type: Sub-task (was: Bug) Parent: YARN-2492 CapacityScheduler should be notified when labels on nodes changed - Key: YARN-2920 URL: https://issues.apache.org/jira/browse/YARN-2920 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Currently, labels on nodes changes will only be handled by RMNodeLabelsManager, but that is not enough upon labels on nodes changes: - Scheduler should be able to do take actions to running containers. (Like kill/preempt/do-nothing) - Used / available capacity in scheduler should be updated for future planning. We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2920: - Target Version/s: 2.7.0 CapacityScheduler should be notified when labels on nodes changed - Key: YARN-2920 URL: https://issues.apache.org/jira/browse/YARN-2920 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2920.1.patch Currently, labels on nodes changes will only be handled by RMNodeLabelsManager, but that is not enough upon labels on nodes changes: - Scheduler should be able to do take actions to running containers. (Like kill/preempt/do-nothing) - Used / available capacity in scheduler should be updated for future planning. We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2920) CapacityScheduler should be notified when labels on nodes changed
[ https://issues.apache.org/jira/browse/YARN-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2920: - Attachment: YARN-2920.1.patch Attached ver.1 patch for this CapacityScheduler should be notified when labels on nodes changed - Key: YARN-2920 URL: https://issues.apache.org/jira/browse/YARN-2920 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-2920.1.patch Currently, labels on nodes changes will only be handled by RMNodeLabelsManager, but that is not enough upon labels on nodes changes: - Scheduler should be able to do take actions to running containers. (Like kill/preempt/do-nothing) - Used / available capacity in scheduler should be updated for future planning. We need add a new event to pass such updates to scheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
