[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255050#comment-15255050
]
Jian He commented on YARN-4390:
-------------------------------
some comments after scanning the patch:
- Looks like the approach is to loop over almost all containers several times
on the cluster for every preemption cycle (3 secs by default), to see
whether some containers can be preempted to make room for the reserved
container on the same node. Will this cause too much overhead in a large
cluster where we have a large amount of containers ? A unit test may test the
time cost for this mega loop.
- unnecessary line breakers are added in multiple places, could you clean those
up ? especially PreemptableResourceCalculator class.
- Does this equal to node.getUnallocatedResource?
{code}
for (RMContainer c : sortedRunningContainers) {
Resources.subtractFrom(available, c.getAllocatedResource());
}
{code}
- Insn't FifoCandidatesSelector the first selector and the selectedCandidates
is empty ?
{code}
// Previous selectors (with higher priority) could have already
// selected containers. We need to deduct preemptable resources
// based on already selected candidates.
CapacitySchedulerPreemptionUtils
.deductPreemptableResourcesBasedSelectedCandidates(preemptionContext,
selectedCandidates);
{code}
> Do surgical preemption based on reserved container in CapacityScheduler
> -----------------------------------------------------------------------
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Affects Versions: 3.0.0, 2.8.0, 2.7.3
> Reporter: Eric Payne
> Assignee: Wangda Tan
> Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf,
> YARN-4390.1.patch, YARN-4390.2.patch, YARN-4390.3.branch-2.patch,
> YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt
> containers. One is that an app could be requesting a large container (say
> 8-GB), and the preemption monitor could conceivably preempt multiple
> containers (say 8, 1-GB containers) in order to fill the large container
> request. These smaller containers would then be rejected by the requesting AM
> and potentially given right back to the preempted app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
