[
https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-4108:
-----------------------------
Attachment: YARN-4108-design-doc-V3.pdf
Uploaded a new doc/poc patch, the proposal of this doc is more conservative,
highlights of the doc:
*Steps*
- Leave existing PCPP as-is, it computes ideal share of each queue, and select
to-be-preempt containers like before.
- When a container expires kill-wait (could be killed), “kill container”
message will be sent to scheduler, and scheduler marks container to be
“killable”
- When scheduler trying to allocate an under-satisfied queue, it will try to
deduct resources of “killable” containers. If a new container could be
allocated/reserved after killing these “killable” containers. Scheduler will
pull the trigger immediately.
In short, this proposal avoids awkwardness that preempted resources cannot be
used by who needs the resource, and then such preempted resources come back to
over-utilized queue again.
More details please refer to doc/WIP-patch. Please feel free to let me know if
you have any comments.
> CapacityScheduler: Improve preemption to preempt only those containers that
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>
> Key: YARN-4108
> URL: https://issues.apache.org/jira/browse/YARN-4108
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-4108-design-doc-V3.pdf,
> YARN-4108-design-doc-v1.pdf, YARN-4108-design-doc-v2.pdf,
> YARN-4108.poc.1.patch
>
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113),
> cross applicaiton preemption (such as priority-based (YARN-1963) /
> fairness-based (YARN-3319)).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
