[
https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098526#comment-15098526
]
Sunil G commented on YARN-4108:
-------------------------------
Thank you [~leftnoteasy] for the detailed updated design doc. Few comments or
queries
1. In current PCPP, we first preempt all reserved continers from all
applications in a queue if its overallocated. After introduding "killable
containers", so we will have some containers which are reserved or running. Am
I correct? or reserved containers are planning to handled differently?
Possible cases:
- queueA's appA has made a reservation on node1. now queueB has demand,
and we can unreserve appA's container requests from node1 and queueB's app can
allocate some containers there.
- or queueB's app can now reserve a container there. This case I think
is explained by the design doc.
Basically we can still try to preempt reserved container first.
2. If I understood correctly, "killable containers" will be triggered with
preeempt event only if a proper allocation can happen for target application
(from underserving queue).
- So do we send preempt_container event to AM here at this point? This
can make a delay of 15secs, so if by some chance scheduling footprint is
changed (some other NMs freed space), we may some overkill. (I guess this can
happen now also).
- 15secs later, RM do forcekill to these containers. Is there any
change for this approach in this new design?
because as per doc, its mentioned that we need to preempt only if we can
reserve. So I am slightly confused here.
3. To cancel "killable container", i think PCPP will take the call by waiting
for some interval. So some new configuration is needed for this?
4. I would like to have some freedom in selecting conatiners (marking) for
preemption. A simple sorting based on submission time or priority seems limited
approach. Could we have some interface here so that we can plugin user specific
comparision cases.
- submission time
- priority
- demand based etc may be
> CapacityScheduler: Improve preemption to preempt only those containers that
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>
> Key: YARN-4108
> URL: https://issues.apache.org/jira/browse/YARN-4108
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-4108-design-doc-V3.pdf,
> YARN-4108-design-doc-v1.pdf, YARN-4108-design-doc-v2.pdf,
> YARN-4108.poc.1.patch, YARN-4108.poc.2-WIP.patch
>
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113),
> cross applicaiton preemption (such as priority-based (YARN-1963) /
> fairness-based (YARN-3319)).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
