[
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576235#comment-15576235
]
Eric Payne commented on YARN-2009:
----------------------------------
[~sunilg], I have one concern about how {{TempAppPerPartition#toBePreempted}}
is calculated. Please consider the use case of where one low priority app is
taking up all resources and a second high priority app is trying to get started:
_NOTE; I am ignoring {{userLimitResource}} and {{idealAssignedForUser}} because
in this use case there is only 1 user. Ignoring {{app.selected}} as well._
{code:title=FifoIntraQueuePreemptionPlugin simplified pseudo code}
calculateIdealAssignedResourcePerApp:
for each app {
app.idealAssigned = min((app.totalUsed + app.pending), queueTotalUnassigned)
}
...
calculateToBePreemptedResourcePerApp:
for each app {
app.toBePreempted = app.totalUsed - app.idealAssigned - app.AMUsed
}
{code}
For this use case, let's consider that all containers, including the AMs, will
be 512 MB. {{app1}} is running, consuming all resources in the queue. The RM
creates {{app2}} and sets its pending value to be 512 MB (for the AM).
Intra-queue preemption does the following during each round:
||AppID||priority||totalUsed MB||AMUsed MB||pending MB||queueTotalUnassigned
MB||
|2|2|0|0|512|12288|
{noformat:title=FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp}
app2.idealAssigned = min((app2.totalUsed + app2.pending), queueTotalUnassigned)
= min((0 + 512), 12288)
= 512
queueTotalUnassigned = queueTotalUnassigned - app2.idealAssigned
= 12288 - 512
= 11776
{noformat}
||AppID||priority||totalUsed MB||AMUsed MB||pending MB||queueTotalUnassigned
MB||
|1|1|12288|512|3584|11776|
{noformat:title=FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp}
app1.idealAssigned = min((app1.totalUsed + app1.pending), queueTotalUnassigned)
= min((12288 + 3584), 11776)
= 11776
queueTotalUnassigned = queueTotalUnassigned - app2.idealAssigned
= 11776 - 11776
= 0
{noformat}
{noformat:title=FifoIntraQueuePreemptionPlugin#calculateToBePreemptedResourcePerApp}
app1.toBePreempted = app1.totalUsed - app1.idealAssigned - app1.AMUsed
= 12288 - 11776 - 512
= 0
{noformat}
In this case, {{app1.toBePreempted}} should be 512 instead of 0. Since it
remains 0, no container ever gets preempted and {{app2}} can never get it's AM
container.
I'm not yet ready to suggest a solution, but I wanted to point this out while
we work on it.
> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> -----------------------------------------------------------------------
>
> Key: YARN-2009
> URL: https://issues.apache.org/jira/browse/YARN-2009
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
> Reporter: Devaraj K
> Assignee: Sunil G
> Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch,
> YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch,
> YARN-2009.0006.patch, YARN-2009.0007.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need
> to consider preempting the low priority application containers first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
