[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429644#comment-15429644
]
Sunil G commented on YARN-4945:
-------------------------------
bq.My assertion is that regardless of what containers are already in the
selectedCandidates list, the intra-queue preemption policy would always need to
select more.
yes, I also meant that same. There are chances that intra-queue preemption
logic may select a container whihc is already selected. So we will continue and
deduct from intra-queue resourceToObtain and will continue. I added this point
to emphasize that intra queue logic will not do anything to already selected
container.
bq.we may want to consider intra-queue preemption configs for dead zone,
natural completion,
Make sense. i will add this point
bq.Is this step calculating the total of preemptable resources for apps in this
queue, per partition?
When we consider resource distribution in a queue, there can be resource over
subscription consider the fact that there were no demand at that time when
these resource were allocated to queue/app. Later at a point , few more apps
came in and caused resource distribution variation based on priority or
user-limit. In such cases, we will be considering priority and user-limit as
separate.
- priority : all pending resource requests for this app will become “resource
to obtain for this app”
- user-limit: may be partial or full pending resource request will become
“resource to obtain for this app”. This is depending on “user-limit_headroom -
current_used”. This much can be considered as demand from this app.
I used pending because of the notion from scheduler. But in preemption world,
that will be mapped to resourceTo Obtain.
And yes, we consider this resourceToObtain per partition level and all
calculations are done as per same.
bq.Is this saying that, when marking containers for preemption, if an app is
under its user limit percent, its containers will not be marked?
I can clarify this. intra-queue preemption will first calculate
resourceTOObtain from those apps which are of high priority (user-limit: those
apps which are over-subscribing resource which crosses its user-limit-quota at
given instance). From these selected apps, we get how much pending is there
and thus will contribute as resourceToObtain (user-limit: in this case, we find
those apps which are starving and not getting its user-limit-quota).
IN these cases, we will come across apps which is already met / more than its
user-limit quota (for priority). So these apps will be skipped and it will be
attribute to resourceToObtain.
bq.Perhaps these should be totally separate policies.
My idea is to come with IntraQueue framework and apply policies like priority
and user-limit on top of that. So with this poc, i am coming with framework and
priority preemption. user-limit can be be added as new policy on top of this
framework. And it will be having the points which are mentioned by you. However
for doc, it will be goof if we could have it common for priority and
user-limit. And we can add the point which you have given in comment to doc.
This will give a better insight for intra-queue preemption. Thoughts?
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: IntraQueuepreemption-CapacityScheduler (Design).pdf,
> YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
