[
https://issues.apache.org/jira/browse/YARN-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14253068#comment-14253068
]
Wangda Tan commented on YARN-2933:
----------------------------------
Hi [~mayank_bansal],
Thanks for working on this,
I think ideal_allocation need to be updated as well, since now we only consider
capacity without labels. We need compute ideal_allocation only in nodes without
labels. More specifically, in {{cloneQueues}}:
{code}
Resource current = Resources.multiply(clusterResources, absUsed);
Resource guaranteed = Resources.multiply(clusterResources, absCap);
Resource maxCapacity = Resources.multiply(clusterResources, absMaxCap);
{code}
Should use totalResourceWIthoutLabel instead of clusterResource here. You can
use {{RMNodeLabelsManager.getResourceByLabel(CommonNodeLabelsManager.NO_LABEL,
clusterResource)}} to get totalResourceWithoutLabel.
I think a simple way to do this might be overwrite clusterResource in
editSchedule, make it to:
{code}
@Override
public void editSchedule(){
CSQueue root = scheduler.getRootQueue();
Resource clusterResources =
- Resources.clone(scheduler.getClusterResource());
+ RMNodeLabelsManager.getResourceByLabel(...)
containerBasedPreemptOrKill(root, clusterResources);
}
{code}
Does this make sense to you?
> Capacity Scheduler preemption policy should only consider capacity without
> labels temporarily
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-2933
> URL: https://issues.apache.org/jira/browse/YARN-2933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
> Reporter: Wangda Tan
> Assignee: Mayank Bansal
> Attachments: YARN-2933-1.patch
>
>
> Currently, we have capacity enforcement on each queue for each label in
> CapacityScheduler, but we don't have preemption policy to support that.
> YARN-2498 is targeting to support preemption respect node labels, but we have
> some gaps in code base, like queues/FiCaScheduler should be able to get
> usedResource/pendingResource, etc. by label. These items potentially need to
> refactor CS which we need spend some time carefully think about.
> For now, what immediately we can do is allow calculate ideal_allocation and
> preempt containers only for resources on nodes without labels, to avoid
> regression like: A cluster has some nodes with labels and some not, assume
> queueA isn't satisfied for resource without label, but for now, preemption
> policy may preempt resource from nodes with labels for queueA, that is not
> correct.
> Again, it is just a short-term enhancement, YARN-2498 will consider
> preemption respecting node-labels for Capacity Scheduler which is our final
> target.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
