[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453773#comment-15453773
]
Wangda Tan commented on YARN-4945:
----------------------------------
Hi Sunil,
Thanks for update, I haven't dig into too many details of this patch yet, I
majorly looked at interactions between preemptable-resource-calculator and
candidate-selector.
Some comments so far:
1) What is intraQueuePreemptionCost? And what is AppPriorityComparator?
2) IntraQueuePreemptableResourceCalculator:
2.1) For the queue-level ideal allocation, is it enough to trust result from
previous policies? PreemptableResourceCalculator saves per-queue ideal
allocation to PCPP#queueToPartitions. I think we don't need add extra logic in
the IntraQueuePreemptableResourceCalculator to resursively calculate it.
Correct?
2.2 computeIntraQueuePreemptionDemand:
- {{if (tq.intraQueuePreemptionCalculationDone == true)}}, is this always false?
- Should we lock the queue inside {{for (LeafQueue leafQUeue : queues)}}?
- return value of getResourceDemandFromAppsPerQueue is not consumed by anyone.
- TempAppPerQueue is not used by anyone
Since the logic looks incompleted yet, here's some thoughts about
implementation/overall-code-structure from my side, hope it may help you.
{code}
IntraQueuePreemptableResourceCalculator {
1. Uses ideal resource calculated by previous policies.
2. Compute per-app ideal/preemptable resource according to per-queue
policies. (Stored in TempAppPerPartition)
}
IntraQueueCandidateSelector {
1. Invoke IntraQueueCandidateSelector to calculate ideal/preemptable of
apps
2. Use
CapacitySchedulerPreemptionUtils.deductPreemptableResourcesBasedSelectedCandidates
to deduct preemptable resource for already selected containers.
for (leafqueue from most underserved) {
for (apps in reserve order) {
if (app.preemptable > 0) {
// Select container logic.
}
}
}
}
{code}
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch,
> YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
