[
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951222#comment-15951222
]
Eric Payne commented on YARN-2113:
----------------------------------
Thanks [~sunilg] for the updated patch. It looks good in gneral, but I still
have one concern.
With the latest patch, there is still a use case in which containers from
under-served users can be preempted. Please consider the following:
- Queue total resources: 100
- Minimum user limit percent: 50%
|| *APP* || *USER* || *PRIORITY* || *USED* || *PENDING* ||
| app1 | user1 | 1 | 50 | 0 |
| app2 | user2 | 1 | 50 | 0 |
| app3 | user3 | 5 | 0 | 10 |
In words, the MULP is 50%, so both {{user1}} and {{user2}} are at their user
limit. When {{user3}} comes along with an app at a higher priority, the
intra-queue preemption monitor should not preempt from either {{user1}} or
{{user2}}. But it does. In fact, it will preempt as much as needed until
{{user3}} reaches its MULP guarantee. This could allow users to use priorities
to abuse the user limit guarantees. This does not happen when the priority of
{{app3}} is the same or below that of the other users' apps.
I am uploading a temporary test patch that shows this. It has 3 new tests. The
difference between them is only the priority of {{app3}} from {{user3}}. The
one that fails is
{{testNoIntraQueuePreemptionIfBelowUserLimitAndHighPriorityExtraUsers}}. The
other two work as expected.
I had to also modify {{ProportionalCapacityPreemptionPolicyMockFramework.java}}
to take an MULP as a paramter in the {{appsConfig}} rather than allow
{{mockApplications()}} to calculate it.
The file
{{TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt}}
should be applied on top of {{YARN-2113.0004.patch}}.
I also have a few other small comments about the code that I will upload
separately.
> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---------------------------------------------------------------
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Sunil G
> Attachments: YARN-2113.0001.patch, YARN-2113.0002.patch,
> YARN-2113.0003.patch, YARN-2113.0004.patch, YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across
> queues per demand and usage. We should also have user-level preemption within
> a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
