[ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429644#comment-15429644 ]
Sunil G commented on YARN-4945: ------------------------------- bq.My assertion is that regardless of what containers are already in the selectedCandidates list, the intra-queue preemption policy would always need to select more. yes, I also meant that same. There are chances that intra-queue preemption logic may select a container whihc is already selected. So we will continue and deduct from intra-queue resourceToObtain and will continue. I added this point to emphasize that intra queue logic will not do anything to already selected container. bq.we may want to consider intra-queue preemption configs for dead zone, natural completion, Make sense. i will add this point bq.Is this step calculating the total of preemptable resources for apps in this queue, per partition? When we consider resource distribution in a queue, there can be resource over subscription consider the fact that there were no demand at that time when these resource were allocated to queue/app. Later at a point , few more apps came in and caused resource distribution variation based on priority or user-limit. In such cases, we will be considering priority and user-limit as separate. - priority : all pending resource requests for this app will become “resource to obtain for this app” - user-limit: may be partial or full pending resource request will become “resource to obtain for this app”. This is depending on “user-limit_headroom - current_used”. This much can be considered as demand from this app. I used pending because of the notion from scheduler. But in preemption world, that will be mapped to resourceTo Obtain. And yes, we consider this resourceToObtain per partition level and all calculations are done as per same. bq.Is this saying that, when marking containers for preemption, if an app is under its user limit percent, its containers will not be marked? I can clarify this. intra-queue preemption will first calculate resourceTOObtain from those apps which are of high priority (user-limit: those apps which are over-subscribing resource which crosses its user-limit-quota at given instance). From these selected apps, we get how much pending is there and thus will contribute as resourceToObtain (user-limit: in this case, we find those apps which are starving and not getting its user-limit-quota). IN these cases, we will come across apps which is already met / more than its user-limit quota (for priority). So these apps will be skipped and it will be attribute to resourceToObtain. bq.Perhaps these should be totally separate policies. My idea is to come with IntraQueue framework and apply policies like priority and user-limit on top of that. So with this poc, i am coming with framework and priority preemption. user-limit can be be added as new policy on top of this framework. And it will be having the points which are mentioned by you. However for doc, it will be goof if we could have it common for priority and user-limit. And we can add the point which you have given in comment to doc. This will give a better insight for intra-queue preemption. Thoughts? > [Umbrella] Capacity Scheduler Preemption Within a queue > ------------------------------------------------------- > > Key: YARN-4945 > URL: https://issues.apache.org/jira/browse/YARN-4945 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Wangda Tan > Attachments: IntraQueuepreemption-CapacityScheduler (Design).pdf, > YARN-2009-wip.patch > > > This is umbrella ticket to track efforts of preemption within a queue to > support features like: > YARN-2009. YARN-2113. YARN-4781. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org