[
https://issues.apache.org/jira/browse/YARN-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214420#comment-15214420
]
Wangda Tan commented on YARN-4822:
----------------------------------
Hi [~sunilg],
bq. I am not much getting much idea abt the use of having multiple policies.
Bcz if one selected candidate from policy1 can be vetoed by another policy (now
we have only one), will it be tough to explain. Here multi layer filtering is
what happening, will it be fine we can use one policy or a group of selected
policy rather than looping all. Since we do not have a usecase now, i think its
ok. But it will be great if you could share some info on same.
I was planning to add a new selection-policy for reserved containers in
YARN-4390. When multiple policies configured in the chain, rear policy can only
add new candidates instead modify already selected candidates from front
policies. You can take a look at WIP patch of YARN-4390 if you have interests.
The reason of doing this is, we need to select preemption candidates to do
surgical preemption in some cases, so some policies should take precedence and
policy with lower priority should respect already selected candidates.
And such behavior is internal protocol of implemented protocols, I don't want
to make it to be pluggable in short term.
bq. From premption round to round. So do we need to keep this object per
PreemptionCandidatesSelector object. May be a generic one (static util class)
is enough for whole PCPP?
I considered of this, but I think make it to be non-static could be more
flexible, since we could have sharable fields during calculation.
> Refactor existing Preemption Policy of CS for easier adding new approach to
> select preemption candidates
> --------------------------------------------------------------------------------------------------------
>
> Key: YARN-4822
> URL: https://issues.apache.org/jira/browse/YARN-4822
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-4822.1.patch, YARN-4822.2.patch, YARN-4822.3.patch,
> YARN-4822.4.patch, YARN-4822.5.patch
>
>
> Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to
> select candidates to be preempted (based on FIFO order of
> applications/containers). It's not a simple to add new candidate-selection
> logics, such as preemption for large container, intra-queeu fairness/policy,
> etc.
> In this JIRA, I propose to do following changes:
> 1) Cleanup code bases, consolidate current logic into 3 stages:
> - Compute ideal sharing of queues
> - Select to-be-preempt candidates
> - Send preemption/kill events to scheduler
> 2) Add a new interface: {{PreemptionCandidatesSelectionPolicy}} for above
> "select to-be-preempt candidates" part. Move existing how to select
> candidates logics to {{FifoPreemptionCandidatesSelectionPolicy}}.
> 3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a
> chain. Preceding PreemptionCandidatesSelectionPolicy has higher priority to
> select candidates, and later PreemptionCandidatesSelectionPolicy can make
> decisions according to already selected candidates and pre-computed queue
> ideal shares of resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
