[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472235#comment-15472235
]
Eric Payne commented on YARN-4945:
----------------------------------
[~sunilg], thanks again for all of the great work you are doing on this issue.
----
\\
- Separate switches for priority and user-limit-percent preemption?
{{ProportionalCapacityPreemptionPolicy#init}} uses
{{SELECT_CANDIDATES_FOR_INTRAQUEUE_PREEMPTION}} to turn on all intra-queue
preemption policies, but the config property name for
{{SELECT_CANDIDATES_FOR_INTRAQUEUE_PREEMPTION}} is
{{select_based_on_priority_of_applications}}.
I actually would like to have the priority policy and the
minumum-user-limit-percent policy be turned on separately. I'm not sure of the
best way to do that, but our users don't use application priority very much.
Perhaps {{CapacitySchedulerConfiguration}} could have something like:
{code}
/**
* For intra-queue preemption, priority based selector can help to preempt
* containers of lowest priority apps to find resources for high priority
* apps.
*/
public static final String
PREEMPTION_SELECT_INTRAQUEUE_CANDIDATES_BY_APP_PRIORITY =
PREEMPTION_CONFIG_PREFIX + "select_based_on_priority_of_applications";
public static final boolean
DEFAULT_PREEMPTION_SELECT_INTRAQUEUE_CANDIDATES_BY_APP_PRIORITY = false;
/**
* For intra-queue preemption, minimum-user-limit-percent based selector can
* help to preempt containers to ensure users are not starved of their
* guaranteed percentage of a queue.
*/
public static final String
PREEMPTION_SELECT_INTRAQUEUE_CANDIDATES_BY_USER_PERCENT_GUARANTEE =
PREEMPTION_CONFIG_PREFIX + "select_based_on_user_percentage_guarantee";
public static final boolean
DEFAULT_SELECT_INTRAQUEUE_CANDIDATES_BY_USER_PERCENT_GUARANTEE = false;
{code}
And then {{ProportionalCapacityPreemptionPolicy#init}} can turn on intra-queue
preemption if either one is set:
{code}
boolean selectIntraQueuePreemptCandidatesByPriority = csConfig.getBoolean(
CapacitySchedulerConfiguration.PREEMPTION_SELECT_INTRAQUEUE_CANDIDATES_BY_APP_PRIORITY,
CapacitySchedulerConfiguration.DEFAULT_PREEMPTION_SELECT_INTRAQUEUE_CANDIDATES_BY_APP_PRIORITY);
boolean selectIntraQueuePreemptCandidatesByUserPercentGuarantee =
csConfig.getBoolean(
CapacitySchedulerConfiguration.PREEMPTION_SELECT_INTRAQUEUE_CANDIDATES_BY_USER_PERCENT_GUARANTEE,
CapacitySchedulerConfiguration.DEFAULT_SELECT_INTRAQUEUE_CANDIDATES_BY_USER_PERCENT_GUARANTEE);
if (selectIntraQueuePreemptCandidatesByPriority ||
selectIntraQueuePreemptCandidatesByUserPercentGuarantee) {
candidatesSelectionPolicies.add(new IntraQueueCandidatesSelector(this));
}
{code}
Then, in {{IntraQueueCandidatesSelector}} logic could be added to do either one
or both intra-queue preemption policies. What do you think?
\\
\\
----
\\
- Could headroom check allow priority inversion?
{{PriorityIntraQueuePreemptionPolicy#getResourceDemandFromAppsPerQueue}}:
{code}
// Can skip apps which are already crossing user-limit.
// For this, Get the userlimit from scheduler and ensure that app is
// not crossing userlimit here. Such apps can be skipped.
Resource userHeadroom = leafQueue.getUserLimitHeadRoomPerApp(
a1.getFiCaSchedulerApp(), context.getPartitionResource(partition),
partition);
if (Resources.lessThanOrEqual(rc,
context.getPartitionResource(partition), userHeadroom,
Resources.none())) {
continue;
}
{code}
I think this code will allow a priority inversion when a user has apps of
different priorities. For example, in a situation like the following, {{App1}}
from {{User1}} is already taking up all of the resources, so its headroom is 0.
But, since {{App2}} is also from {{User1}}, the above code will never allow
preemption to occur. Is that correct?
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending
Resources||
|QUEUE1|User1|App1|1|200|0|
|QUEUE1|User1|App2|10|0|50|
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch,
> YARN-2009-wip.patch, YARN-2009-wip.v3.patch, YARN-2009.v0.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
