[
https://issues.apache.org/jira/browse/YUNIKORN-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YUNIKORN-3450:
-------------------------------------
Labels: pull-request-available (was: )
> [Core] False preemption shortfall aborts queue preemption and starves large
> tasks despite sufficient guaranteed quota
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3450
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3450
> Project: Apache YuniKorn
> Issue Type: Bug
> Affects Versions: 1.6.0
> Reporter: Hedger Lai
> Assignee: Hedger Lai
> Priority: Major
> Labels: pull-request-available
>
> h2. Relationship to YUNIKORN-3449
> This issue is a companion/follow-up to YUNIKORN-3449:
> * *YUNIKORN-3449* fixes the *Node physical capacity* dimension (where the
> node's existing available capacity was ignored during shortfall verification).
> * *This issue* fixes the *Queue quota* dimension (where the ask queue's
> existing remaining guaranteed headroom is ignored during shortfall
> verification).
> Together, these two issues resolve the false preemption shortfall defects in
> {{TryPreemption()}}.
> h2. 1. The Symptom
> In {{Preemptor.TryPreemption()}}, preemption for large tasks *falsely aborts
> with PreemptionShortfall and starves indefinitely*, even when preempting
> available victims has *already freed enough quota to bring the ask queue
> under its Guaranteed limit*.
> h2. 2. Root Cause
> In {{pkg/scheduler/objects/preemption.go}}:
> {code:go}
> for k, victimVal := range victimsTotalResource.Resources {
> if needVal, ok := p.ask.GetAllocatedResource().Resources[k]; ok &&
> victimVal < needVal {
> hasShortfall = true
> break
> }
> }
> {code}
> This check crudely demands {{victimVal >= needVal}} (victims alone must equal
> the entire ask demand).
> It *completely ignores the ask queue's existing remaining guaranteed
> headroom*.
> *Real-World Example (Pure extraVictims Scenario):*
> * The target node is completely idle and has sufficient physical space (no
> node victims needed). Preemption is triggered purely to satisfy queue limits
> via {{calculateAdditionalVictims()}}.
> * Parent Queue Max = 10. Currently used = 5. *Remaining Headroom = 5*.
> * A large Ask needs 10.
> * *Net Queue Deficit* = {{5 (used) + 10 (ask) - 10 (max) = 5}}.
> * Upstream correctly preempts 1 victim (5 vcores), freeing enough quota so
> {{0 (used) + 10 (ask) = 10 <= 10}}.
> * But downstream checks {{victimVal (5) < needVal (10)}}, falsely triggers
> {{PreemptionShortfall}}, and *aborts preemption!*
> h2. 3. Deterministic Reproduction Unit Test
> This bug reproduces deterministically in {{preemption_test.go}} (0.01s):
> {code:go}
> func TestTryPreemption_QueueResidualShortfallDeficit(t *testing.T) {
> appQueueMapping := NewAppQueueMapping()
> node1 := newNode(nodeID1, map[string]resources.Quantity{"first": 30})
> node2 := newNode(nodeID2, map[string]resources.Quantity{"first": 30})
> iterator := getNodeIteratorFn(node1, node2)
> rootQ, err := createRootQueue(map[string]string{"first": "60"})
> assert.NilError(t, err)
> // Parent Max = 10
> parentQ, err := createManagedQueueGuaranteed(rootQ, "parent", true,
> map[string]string{"first": "10"}, nil, appQueueMapping)
> assert.NilError(t, err)
> _, err = createManagedQueueGuaranteed(parentQ, "child1", false, nil,
> nil, appQueueMapping)
> assert.NilError(t, err)
> childQ2, err := createManagedQueueGuaranteed(parentQ, "child2", false,
> nil, map[string]string{"first": "20"}, appQueueMapping)
> assert.NilError(t, err)
> childQ3, err := createManagedQueueGuaranteed(parentQ, "child3", false,
> nil, nil, appQueueMapping)
> assert.NilError(t, err)
> // Victim alloc4 uses 5 on node2. Parent allocated = 5. Headroom = 10 -
> 5 = 5.
> app3 := newApplication(appID3, "default", "root.parent.child3")
> app3.SetQueue(childQ3)
> childQ3.AddApplication(app3)
> appQueueMapping.AddAppQueueMapping(app3.ApplicationID, childQ3)
> ask4 := newAllocationAsk("alloc4", appID3,
> resources.NewResourceFromMap(map[string]resources.Quantity{"first": 5}))
> assert.NilError(t, app3.AddAllocationAsk(ask4))
> alloc4 := newAllocationWithKey("alloc4", appID3, nodeID2,
> resources.NewResourceFromMap(map[string]resources.Quantity{"first": 5}))
> app3.AddAllocation(alloc4)
> assert.Check(t, node2.TryAddAllocation(alloc4), "node2 alloc4 failed")
> assert.NilError(t,
> childQ3.TryIncAllocatedResource(ask4.GetAllocatedResource()))
> // Preemptor ask needs 10. Deficit is 5. Preempting alloc4 (5)
> satisfies queue quota completely (0 + 10 <= 10).
> app2, ask3, err := creatApp2(childQ2,
> map[string]resources.Quantity{"first": 10}, "alloc3", appQueueMapping)
> assert.NilError(t, err)
> headRoom :=
> resources.NewResourceFromMap(map[string]resources.Quantity{"first": 20})
> preemptor := NewPreemptor(app2, headRoom, 30*time.Second, ask3,
> iterator(), false)
> feasibleNodes := map[string]int{nodeID2: 1}
> plugin := mock.NewPreemptionPredicatePlugin(nil, feasibleNodes, false,
> false)
> plugins.RegisterSchedulerPlugin(plugin)
> defer plugins.UnregisterSchedulerPlugins()
> result, ok := preemptor.TryPreemption()
> assert.Assert(t, ok, "preemption should succeed: queue deficit is 5,
> victim is 5")
> assert.Assert(t, result != nil, "expected non-nil result")
> assert.Equal(t, "alloc3", result.Request.GetAllocationKey())
> }
> {code}
> h2. 4. Terminal Failure Output (Evidence)
> {code:text}
> === RUN TestTryPreemption_QueueResidualShortfallDeficit
> preemption_test.go:2567: assertion failed: ok is false: preemption should
> succeed: queue deficit is 5, victim is 5
> --- FAIL: TestTryPreemption_QueueResidualShortfallDeficit (0.01s)
> FAIL
> {code}
> h2. 5. Proposed Solution
> Update the queue preemption shortfall check to evaluate the net residual
> deficit:
> Verify whether {{remainingGuaranteed + victimsTotalResource >= ask}}, rather
> than requiring {{victimsTotalResource >= ask}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
