[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455570#comment-15455570
]
Eric Payne commented on YARN-4945:
----------------------------------
Thanks very much [~sunilg] and [~leftnoteasy].
{quote}
1.
I think we might need to come with a limit on how much resource can be
preempted from over-utilizing users's apps. WE do have
max-preemption-per-round. But sometimes it may be more as it may be configured
for inter-queue. Since we are sharing this config, i think we can have a config
to limit the preemption for user-limit. For priority, i have considered a
certain limit to control this scenario. Thoughts?
{quote}
I think we do need several intra-queue configs that are separate from the
existing (inter-queue) ones. For inter-queue vs. intra-queue, I think we need a
separate one at least for {{total_preemption_per_round}} and
{{max_ignored_over_capacity}}, and maybe even for
{{natural_termination_factor}} and {{max_wait_before_kill}}.
Are you also suggesting that we need these configs to also be spearate between
user-limit-percent preemption and priority preemption within intra queue? I
don't have a strong opinion either way, but if we can keep all configs the same
between intra-queue preemption policies, I would like to do that, just to avoid
confusion and complication.
bq. I will not consider preemption demand from a high priority if that app is
already crossing the user-limit.
I just want to make sure we are talking about the same thing. In the case I am
worried about, the high priority app is _*not*_ over any limit. There is an
inversion happening because the lower priority app has containers and the high
priority app wants them. But, if the low priority app is from a user that is at
or below its {{minimum-user-limit-percent}}, the higher priority app must not
continue to preempt from the lower priority app. This only can happen when the
two apps are from different users.
{quote}
I think normalization for inter-queue / intra-queue preemption is one of the
top priority goal for this feature.
If you take a look at existing preemption code, it normalizes preempt-able
resource for reserved-container-candidate-selector and fifo-candidate-selector.
We can do the similar normalization for inter/intra-queue preemption.
{quote}
Trying to do this coordination seems to me to be quite complicated. Would it be
sufficient to just avoid preempting during the intra-queue policies if there
are already containers in the {{selectedContainers}} list?
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch,
> YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
