[
https://issues.apache.org/jira/browse/YARN-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216790#comment-15216790
]
Wangda Tan commented on YARN-4822:
----------------------------------
Hi [~sunilg],
bq. In that case, could we mark Interface visibility as Unstable/Evolving for
this CapacitySchedulerPreemptionUtils class since its public?
All preemption-related classes will be used by scheduler internally only, no
external application should use such methods.
bq. Also I am thinking whether we can make CapacitySchedulerPreemptionContext
also visible with a limited scope. I can query some useful read-only interfaces
from Scheduler for metric purpose. May be to get getKillableContainers. I am
still thinking how much information we can fetch from context rather than
directly poking PCPP for some useful metric/log or debuggabillity perspective.
Thoughts?
This is a good point, we can definitely add some methods so UI can benefit from
them. KillableContainers can be either get from scheduler or PCPP, and
preemption candidates only stored in PCPP. We can add method in separate JIRAs
when we have concrete requirements.
bq. In selectCandidates, we skip containers which are already selected. And
also we are checking whether this container is already present in preemptMap
too. This check happens for every container now, may be some optimization can
be planned in next version?
This looks fine to me actually, since check only takes O(1). Do you have any
performance concerns?
bq. If tryPreemptContainerAndDeductResToObtain return false because container
is already present in preemptMap, we can continue try for other container for
same app , correct? Now we are breaking. May be I am wrong, pls help to share
if missed some other factor.
Fixed.
bq. ... AM resource percentage per partition ...
Agree, we can handle it in a separate JIRA.
Fixed all minor nits, except:
bq. In tryPreemptContainerAndDeductResToObtain, could you pls add queuename to
the debug log.
Currently we don't have queueName in RMContainer, I will add it in YARN-4390.
Attaching ver.7 patch.
> Refactor existing Preemption Policy of CS for easier adding new approach to
> select preemption candidates
> --------------------------------------------------------------------------------------------------------
>
> Key: YARN-4822
> URL: https://issues.apache.org/jira/browse/YARN-4822
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-4822.1.patch, YARN-4822.2.patch, YARN-4822.3.patch,
> YARN-4822.4.patch, YARN-4822.5.patch, YARN-4822.6.patch, YARN-4822.7.patch
>
>
> Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to
> select candidates to be preempted (based on FIFO order of
> applications/containers). It's not a simple to add new candidate-selection
> logics, such as preemption for large container, intra-queeu fairness/policy,
> etc.
> In this JIRA, I propose to do following changes:
> 1) Cleanup code bases, consolidate current logic into 3 stages:
> - Compute ideal sharing of queues
> - Select to-be-preempt candidates
> - Send preemption/kill events to scheduler
> 2) Add a new interface: {{PreemptionCandidatesSelectionPolicy}} for above
> "select to-be-preempt candidates" part. Move existing how to select
> candidates logics to {{FifoPreemptionCandidatesSelectionPolicy}}.
> 3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a
> chain. Preceding PreemptionCandidatesSelectionPolicy has higher priority to
> select candidates, and later PreemptionCandidatesSelectionPolicy can make
> decisions according to already selected candidates and pre-computed queue
> ideal shares of resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
