weichen lai created YUNIKORN-3452:
-------------------------------------
Summary: Node Preemption: Fix reversed penalty scores between
originator and opted-out pods
Key: YUNIKORN-3452
URL: https://issues.apache.org/jira/browse/YUNIKORN-3452
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: weichen lai
Assignee: weichen lai
h3. Problem
During preemption, {{checkPreemptionPredicates()}} compares candidate nodes
using {{betterThan()}}, which prefers nodes with lower penalty scores from
{{getSolutionScore()}}. In short: a lower score means a better, less painful
choice.
In YuniKorn, an application driver/originator pod must be protected over an
opted-out worker pod. If a driver pod is killed, the entire application fails
(e.g., Spark, Ray, Flink), while killing an opted-out worker pod
({{allowPreemptSelf: false}}) only affects that single task.
However, the penalty constants in {{pkg/scheduler/objects/preemption.go}} are
currently set backwards:
{code:go}scoreOriginator uint64 = 1 << 33 // 8,589,934,592
scoreNoPreempt uint64 = 1 << 34 // 17,179,869,184
{code}
Because {{scoreOriginator}} is smaller than {{scoreNoPreempt}}, the scheduler
treats killing an originator pod as a "cheaper" option than killing an
opted-out worker pod. In practice, this can cause the scheduler to pick the
wrong node and kill an application driver instead of an opted-out worker.
h3. Reproducer / Unit Test
Here is a simple test that reproduces the issue and fails on current {{master}}:
{code:go}func TestSolutionScoring_OriginatorVsOptedOut(t *testing.T) {
allocMap := map[string][]*Allocation{
"node-opted-out": {allocForScore(false, false)},
"node-originator": {allocForScore(true, true)},
}
solOptedOut := &predicateCheckResult{nodeID: "node-opted-out", success:
true, index: 0}
solOriginator := &predicateCheckResult{nodeID: "node-originator",
success: true, index: 0}
// An originator pod is the driver/core of an app and must be protected
over an opted-out worker pod.
// Therefore, a solution that sacrifices an opted-out pod MUST be
betterThan a solution that sacrifices an originator pod.
assert.Check(t, solOptedOut.betterThan(solOriginator, allocMap),
"sacrificing opted-out pod should be a better solution than
sacrificing originator pod")
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
