[
https://issues.apache.org/jira/browse/YARN-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209469#comment-15209469
]
Wangda Tan commented on YARN-4822:
----------------------------------
[~eepayne], [~sunilg], [~jianhe],
Appreciate if you could take a look at latest patch, it contains a couple of
refactorings:
- PCPP becomes 2 parts:
1) Basic code such as clone queues, record what to preempt and send kill event
when max-wait reaches.
2) Candidates-selection policy, includes calculate ideal allocation and select
preemption candidates
- Original calculate ideal allocation and select preemption candidates goes to
two classes
1) FifoPreemptableAmountCalculator is for ideal allocation calculation
2) FifoCandidatesSelectionPolicy is for how to select containers
- CandidatesSelectionPolicy and calculator needs to read some fields from PCPP,
so I add an interface for them to use, which is implemented by PCPP:
CapacitySchedulerPreemptionContext
- Moved all configurations keys from PCPP to CapacitySchedulerConfiguration, so
admin can set configurations in either yarn-site.xml or capacity-scheduler.xml.
(Ideally should be set in capacity-scheduler.xml, however, existing user sets
configs in yarn-site.xml. Since CapacitySchedulerConfiguration reads
yarn-site.xml as well, it is backward compatible change.)
Thanks,
> Refactor existing Preemption Policy of CS for easier adding new approach to
> select preemption candidates
> --------------------------------------------------------------------------------------------------------
>
> Key: YARN-4822
> URL: https://issues.apache.org/jira/browse/YARN-4822
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: YARN-4822.1.patch, YARN-4822.2.patch, YARN-4822.3.patch,
> YARN-4822.4.patch
>
>
> Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to
> select candidates to be preempted (based on FIFO order of
> applications/containers). It's not a simple to add new candidate-selection
> logics, such as preemption for large container, intra-queeu fairness/policy,
> etc.
> In this JIRA, I propose to do following changes:
> 1) Cleanup code bases, consolidate current logic into 3 stages:
> - Compute ideal sharing of queues
> - Select to-be-preempt candidates
> - Send preemption/kill events to scheduler
> 2) Add a new interface: {{PreemptionCandidatesSelectionPolicy}} for above
> "select to-be-preempt candidates" part. Move existing how to select
> candidates logics to {{FifoPreemptionCandidatesSelectionPolicy}}.
> 3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a
> chain. Preceding PreemptionCandidatesSelectionPolicy has higher priority to
> select candidates, and later PreemptionCandidatesSelectionPolicy can make
> decisions according to already selected candidates and pre-computed queue
> ideal shares of resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
