[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15471536#comment-15471536
]
Eric Payne commented on YARN-4945:
----------------------------------
[~leftnoteasy] and [~sunilg], I'm still concerned about having the intra-queue
preemption policies adding containers to the {{selectedCandidates}} list if the
inter-queue policies have already added containers. In that case, the
containers selected by the intra-queue policies may not go back to the correct
queue. Consider this use case:
Queues (all are preemptable):
||Queue Name||Guaranteed Resources||Max
Resources||{{total_preemption_per_round}}||
|root|200|200|0.1|
|QUEUE1|100|200|0.1|
|QUEUE2|100|200|0.1|
# {{User1}} starts {{App1}} on {{QUEUE1}} and uses all 200 resources. These
containers are long-running and will not be released any time soon:
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending
Resources||Selected For Preemption||
|QUEUE1|User1|App1|1|200|0|0|
# {{User2}} starts {{App2}} on {{QUEUE2}} and requests 100 resources:
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending
Resources||Selected For Preemption||
|QUEUE1|User1|App1|1|200|0|0|
|QUEUE2|User2|App2|1|0|100|0|
# {{User1}} starts {{App3}} at a high priority on {{QUEUE1}} and requests 50
resources:
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending
Resources||Selected For Preemption||
|QUEUE1|User1|App1|1|200|0|0|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|0|100|0|
# Since {{total_preemption_per_round}} is 0.1, only 10% of the needed resources
will be selected per round. So, the inter-queue preemption policies select 10
resources to be preempted from {{App1}}.
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending
Resources||Selected For Preemption||
|QUEUE1|User1|App1|1|200|0|10|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|0|100|0|
# Then, the priority-intra-queue preemption policy selects 5 more resources to
be preempted from {{App1}}.
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending
Resources||Selected For Preemption||
|QUEUE1|User1|App1|1|200|0|15|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|0|100|0|
# At this point, 15 resources are preempted from {{App1}}.
# Since {{QUEUE2}} is asking for 100 resources, and is extremely underserved
(from an inter-queue point of view), the capacity scheduler gives all 15
resources to {{QUEUE2}}, and the priority inversion remains in {{QUEUE1}}.
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending
Resources||Selected For Preemption||
|QUEUE1|User1|App1|1|185|15|0|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|15|85|0|
This is why I am concerned that when containers are already selected by the
inter-queue preemption policies, it may not be beneficial to have the
intra-queue policies preempt containers as well.
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch,
> YARN-2009-wip.patch, YARN-2009-wip.v3.patch, YARN-2009.v0.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
