[
https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741974#comment-14741974
]
Arun Suresh commented on YARN-4120:
-----------------------------------
Currently, pre-emption happens in two passes:
1) Iterate through the leaf Queues, find resource deficits (sum of resources to
be preempted to from other apps to allow apps below share to run)
2) iterate through Schedulables and pre-empt enough containers to match the
resource collected in 1.
One goal of YARN-2154 (when its ready) is to try to address the issue brought
up here, wherein, instead of asking a root queue to find Schedulables within
its hierarchy that can pre-empt containers, it tries to match an actual
resource ask (by an app below fair share) with containers from apps (above fair
share).
I believe the above logic might solve both issues raised here.. thoughts ?
> FSAppAttempt.getResourceUsage() should not take preemptedResource into account
> ------------------------------------------------------------------------------
>
> Key: YARN-4120
> URL: https://issues.apache.org/jira/browse/YARN-4120
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
> Reporter: Xianyin Xin
>
> When compute resource usage for Schedulables, the following code is envolved,
> {{FSAppAttempt.getResourceUsage}},
> {code}
> public Resource getResourceUsage() {
> return Resources.subtract(getCurrentConsumption(), getPreemptedResources());
> }
> {code}
> and this value is aggregated to FSLeafQueues and FSParentQueues. In my
> opinion, taking {{preemptedResource}} into account here is not reasonable,
> there are two main reasons,
> # it is something in future, i.e., even though these resources are marked as
> preempted, it is currently used by app, and these resources will be
> subtracted from {{currentCosumption}} once the preemption is finished. it's
> not reasonable to make arrange for it ahead of time.
> # there's another problem here, consider following case,
> {code}
> root
> / \
> queue1 queue2
> / \
> queue1.3, queue1.4
> {code}
> suppose queue1.3 need resource and it can preempt resources from queue1.4,
> the preemption happens in the interior of queue1. But when compute resource
> usage of queue1, {{queue1.resourceUsage = it's_current_resource_usage -
> preemption}} according to the current code, which is unfair to queue2 when
> doing resource allocating.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
