[
https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051332#comment-15051332
]
Sunil G commented on YARN-4108:
-------------------------------
Hi [~leftnoteasy]
Thank you for sharing the detailed doc and patch. Its a wonderful effort and
came out nicely. I went through the doc mainly, and checked few part of the
patch. I will got through patch in detail soon and share more comments if I
have.
Few major doubts:
1. With different {{PreemptionType}}, are we planning to handle preemption
across queue, within queue (fifo/priority), within user etc? YARN-2009 was
trying to handling the preemption within a queue adhering to priority.
2. Currently all containers from a node is selected and tried to find which all
are matching the preemption type. Later {{selectContainersToPreempt}} helps to
clear out the non-valid containers.
>> I would see it will be a great help if flexibility is provided with some
>> interface to sort containers eventhough a great deal of validation is done.
Sorting parameter can be
- submitted time
- priority of app (since we take bunch of containers from a node first,
only few apps in cluster will come in one shot)
- priority of containers
- time remaining for the container to finish (% of completion)
With these flexibility, user can tune which containers will be his first choice
for preemption provided all the size/user limit/locality are matched.
3. Could we get a choice to kill container based on data locality, I could see
the changes but cudnt see how its achieved in preemption manager end.
> CapacityScheduler: Improve preemption to preempt only those containers that
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>
> Key: YARN-4108
> URL: https://issues.apache.org/jira/browse/YARN-4108
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-4108-design-doc-v1.pdf, YARN-4108.poc.1.patch
>
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113),
> cross applicaiton preemption (such as priority-based (YARN-1963) /
> fairness-based (YARN-3319)).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
