Xianyin Xin commented on YARN-4120:

hi [~asuresh], thanks for your comment. I've go through YARN-2154, i believe it 
is a nice solution for the problems of current preemption logic. But i think 
the current patch of YARN-2154 could not solve the issue raised in this jira 
(please correct me if i wrongly understood YARN-2154.). We should distinguish 
{{usage}} and {{usage - preemption}} in {{getResourceUsgae}}, because 
{{getResourceUsage}} is used both by the preemption logic and the resource 
allocation logic. Of course we can consider this in the new implemention in 
YARN-2154 and solve them together.

> FSAppAttempt.getResourceUsage() should not take preemptedResource into account
> ------------------------------------------------------------------------------
>                 Key: YARN-4120
>                 URL: https://issues.apache.org/jira/browse/YARN-4120
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>            Reporter: Xianyin Xin
> When compute resource usage for Schedulables, the following code is envolved,
> {{FSAppAttempt.getResourceUsage}},
> {code}
> public Resource getResourceUsage() {
>   return Resources.subtract(getCurrentConsumption(), getPreemptedResources());
> }
> {code}
> and this value is aggregated to FSLeafQueues and FSParentQueues. In my 
> opinion, taking {{preemptedResource}} into account here is not reasonable, 
> there are two main reasons,
> # it is something in future, i.e., even though these resources are marked as 
> preempted, it is currently used by app, and these resources will be 
> subtracted from {{currentCosumption}} once the preemption is finished. it's 
> not reasonable to make arrange for it ahead of time. 
> # there's another problem here, consider following case,
> {code}
>             root
>            /    \
>       queue1   queue2
>       /    \
> queue1.3, queue1.4
> {code}
> suppose queue1.3 need resource and it can preempt resources from queue1.4, 
> the preemption happens in the interior of queue1. But when compute resource 
> usage of queue1, {{queue1.resourceUsage = it's_current_resource_usage - 
> preemption}} according to the current code, which is unfair to queue2 when 
> doing resource allocating.

This message was sent by Atlassian JIRA

Reply via email to