Hedger Lai created YUNIKORN-3450:
------------------------------------
Summary: [Core] False preemption shortfall aborts queue preemption
and starves large tasks despite sufficient guaranteed quota
Key: YUNIKORN-3450
URL: https://issues.apache.org/jira/browse/YUNIKORN-3450
Project: Apache YuniKorn
Issue Type: Bug
Affects Versions: 1.6.0
Reporter: Hedger Lai
Assignee: Hedger Lai
h2. Relationship to YUNIKORN-3449
This issue is a companion/follow-up to YUNIKORN-3449:
* *YUNIKORN-3449* fixes the *Node physical capacity* dimension (where the
node's existing available capacity was ignored during shortfall verification).
* *This issue* fixes the *Queue quota* dimension (where the ask queue's
existing remaining guaranteed headroom is ignored during shortfall
verification).
Together, these two issues resolve the false preemption shortfall defects in
{{TryPreemption()}}.
h2. 1. The Symptom
In {{Preemptor.TryPreemption()}}, preemption for large tasks *falsely aborts
with PreemptionShortfall and starves indefinitely*, even when preempting
available victims has *already freed enough quota to bring the ask queue under
its Guaranteed limit*.
h2. 2. Root Cause
In {{pkg/scheduler/objects/preemption.go}}:
{code:go}
for k, victimVal := range victimsTotalResource.Resources {
if needVal, ok := p.ask.GetAllocatedResource().Resources[k]; ok &&
victimVal < needVal {
hasShortfall = true
break
}
}
{code}
This check crudely demands {{victimVal >= needVal}} (victims alone must equal
the entire ask demand).
It *completely ignores the ask queue's existing remaining guaranteed headroom*.
*Real-World Example (Pure extraVictims Scenario):*
* The target node is completely idle and has sufficient physical space (no node
victims needed). Preemption is triggered purely to satisfy queue limits via
{{calculateAdditionalVictims()}}.
* Parent Queue Max = 10. Currently used = 5. *Remaining Headroom = 5*.
* A large Ask needs 10.
* *Net Queue Deficit* = {{5 (used) + 10 (ask) - 10 (max) = 5}}.
* Upstream correctly preempts 1 victim (5 vcores), freeing enough quota so {{0
(used) + 10 (ask) = 10 <= 10}}.
* But downstream checks {{victimVal (5) < needVal (10)}}, falsely triggers
{{PreemptionShortfall}}, and *aborts preemption!*
h2. 3. Deterministic Reproduction Unit Test
This bug reproduces deterministically in {{preemption_test.go}} (0.01s):
{code:go}
func TestTryPreemption_QueueResidualShortfallDeficit(t *testing.T) {
appQueueMapping := NewAppQueueMapping()
node1 := newNode(nodeID1, map[string]resources.Quantity{"first": 30})
node2 := newNode(nodeID2, map[string]resources.Quantity{"first": 30})
iterator := getNodeIteratorFn(node1, node2)
rootQ, err := createRootQueue(map[string]string{"first": "60"})
assert.NilError(t, err)
// Parent Max = 10
parentQ, err := createManagedQueueGuaranteed(rootQ, "parent", true,
map[string]string{"first": "10"}, nil, appQueueMapping)
assert.NilError(t, err)
_, err = createManagedQueueGuaranteed(parentQ, "child1", false, nil,
nil, appQueueMapping)
assert.NilError(t, err)
childQ2, err := createManagedQueueGuaranteed(parentQ, "child2", false,
nil, map[string]string{"first": "20"}, appQueueMapping)
assert.NilError(t, err)
childQ3, err := createManagedQueueGuaranteed(parentQ, "child3", false,
nil, nil, appQueueMapping)
assert.NilError(t, err)
// Victim alloc4 uses 5 on node2. Parent allocated = 5. Headroom = 10 -
5 = 5.
app3 := newApplication(appID3, "default", "root.parent.child3")
app3.SetQueue(childQ3)
childQ3.AddApplication(app3)
appQueueMapping.AddAppQueueMapping(app3.ApplicationID, childQ3)
ask4 := newAllocationAsk("alloc4", appID3,
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 5}))
assert.NilError(t, app3.AddAllocationAsk(ask4))
alloc4 := newAllocationWithKey("alloc4", appID3, nodeID2,
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 5}))
app3.AddAllocation(alloc4)
assert.Check(t, node2.TryAddAllocation(alloc4), "node2 alloc4 failed")
assert.NilError(t,
childQ3.TryIncAllocatedResource(ask4.GetAllocatedResource()))
// Preemptor ask needs 10. Deficit is 5. Preempting alloc4 (5)
satisfies queue quota completely (0 + 10 <= 10).
app2, ask3, err := creatApp2(childQ2,
map[string]resources.Quantity{"first": 10}, "alloc3", appQueueMapping)
assert.NilError(t, err)
headRoom :=
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 20})
preemptor := NewPreemptor(app2, headRoom, 30*time.Second, ask3,
iterator(), false)
feasibleNodes := map[string]int{nodeID2: 1}
plugin := mock.NewPreemptionPredicatePlugin(nil, feasibleNodes, false,
false)
plugins.RegisterSchedulerPlugin(plugin)
defer plugins.UnregisterSchedulerPlugins()
result, ok := preemptor.TryPreemption()
assert.Assert(t, ok, "preemption should succeed: queue deficit is 5,
victim is 5")
assert.Assert(t, result != nil, "expected non-nil result")
assert.Equal(t, "alloc3", result.Request.GetAllocationKey())
}
{code}
h2. 4. Terminal Failure Output (Evidence)
{code:text}
=== RUN TestTryPreemption_QueueResidualShortfallDeficit
preemption_test.go:2567: assertion failed: ok is false: preemption should
succeed: queue deficit is 5, victim is 5
--- FAIL: TestTryPreemption_QueueResidualShortfallDeficit (0.01s)
FAIL
{code}
h2. 5. Proposed Solution
Update the queue preemption shortfall check to evaluate the net residual
deficit:
Verify whether {{remainingGuaranteed + victimsTotalResource >= ask}}, rather
than requiring {{victimsTotalResource >= ask}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
