Hedger Lai created YUNIKORN-3449:
------------------------------------
Summary: [Core] Preemption falsely aborts and starves large asks
by ignoring node available capacity in shortfall check
Key: YUNIKORN-3449
URL: https://issues.apache.org/jira/browse/YUNIKORN-3449
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Affects Versions: 1.6.0
Reporter: Hedger Lai
Assignee: Hedger Lai
h3. Problem Summary
In {{Preemptor.TryPreemption()}}, when selecting preemption victims on a
candidate node, upstream {{tryNodes()}} and {{calculateVictimsByNode()}}
evaluate victims by adding their resources to {{nodeAvailable}}:
{code:go}
nodeCurrentAvailable.AddTo(victim.GetAllocatedResource())
if nodeCurrentAvailable.FitIn(p.ask.GetAllocatedResource()) && index < 0 {
index = len(results)
}
{code}
Upstream correctly stops collecting victims once {{nodeAvailable + victims >=
ask}}. That is, upstream only selects victims to cover the node's residual
*deficit* ({{ask - nodeAvailable}}).
However, downstream in {{TryPreemption()}} (lines 670-678 on master), the
shortfall validation compares {{victimsTotalResource}} directly against the
full ask requirement ({{p.ask.GetAllocatedResource()}}):
{code:go}
hasShortfall := victimsTotalResource.IsEmpty()
if !hasShortfall {
for k, victimVal := range victimsTotalResource.Resources {
if needVal, ok := p.ask.GetAllocatedResource().Resources[k]; ok &&
victimVal < needVal {
hasShortfall = true
break
}
}
}
if hasShortfall {
// there is shortfall, so preemption doesn't help
p.ask.LogAllocationFailure(common.PreemptionShortfall, true)
return nil, false
}
{code}
Because {{victimsTotalResource}} only accumulates victim allocations without
factoring in the node's existing available capacity
({{p.nodeAvailableMap[nodeID]}}), preemption is falsely aborted whenever a
candidate node has partial available capacity!
h3. Reproduction Scenario
Consider a node with 4 vcores capacity:
# *Node available capacity*: 2 vcores idle ({{nodeAvailable = 2}}).
# *Running allocation*: Pod A using 2 vcores (candidate victim).
# *Pending Ask*: Requires 4 vcores.
* *Upstream behavior*:
{{calculateVictimsByNode()}} starts with 2 free vcores. It selects Pod A (2
vcores). {{nodeCurrentAvailable}} becomes {{2 + 2 = 4 vcores}}, which fits the
ask. Upstream returns {{nodeVictims = [Pod A]}} (2 vcores).
* *Downstream failure*:
{{victimsTotalResource}} = 2 vcores.
Downstream evaluates {{victimVal (2) < needVal (4)}} -> {{hasShortfall = true}}.
{{TryPreemption()}} aborts with {{PreemptionShortfall}}, even though preempting
Pod A would free up the 4 vcores needed to schedule the ask!
h3. Root Cause
In {{TryPreemption()}}, the shortfall check assumes that the total resources of
all selected victims alone must equal or exceed the ask's resource requirement.
It completely ignores that the selected node already contributes
{{p.nodeAvailableMap[nodeID]}} towards fulfilling the ask.
h3. Proposed Solution
When checking for preemption shortfall on the chosen node:
# Factor in the node's existing available capacity:
The required physical capacity from victims on the node is {{deficit = max(0,
ask - nodeAvailable)}}.
# Ensure the shortfall check verifies whether victims satisfy the node deficit
(for physical node capacity) as well as any queue headroom requirements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
