[
https://issues.apache.org/jira/browse/YUNIKORN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weichen lai updated YUNIKORN-3445:
----------------------------------
Description:
h3. Overview
In {{{}pkg/scheduler/objects/preemption_utilities.go{}}},
{{SortAllocationsBasedOnAsk()}} and {{scoreAllocationBasedOnAsk()}} determine
victim ordering during preemption.
Under the current implementation, allocations with {{!allowPreemptSelf}}
(opted-out) or {{isOriginator}} (driver/manager) receive higher-order score
bits ({{{}scoreNoPreempt{}}} = {{{}1 << 34{}}}, {{scoreOriginator}} = {{{}1 <<
33{}}}):
{code:go}
func scoreAllocationBasedOnAsk(allocation *Allocation, ask *resources.Resource)
uint64 {
var score uint64 = 0
if allocation.IsOriginator() {
score |= scoreOriginator // 1 << 33
}
if !allocation.IsAllowPreemptSelf() {
score |= scoreNoPreempt // 1 << 34
}
score += allocation.GetAllocatedResource().TypeMatching(ask)
return score
}
{code}
Because {{SortAllocationsBasedOnAsk()}} orders allocations in descending order
({{{}scoreLeft > scoreRight{}}}), opted-out and originator allocations receive
higher scores and are placed at the beginning of the candidate slice rather
than at the end.
h3. Comparison with Design Documentation
This ordering conflicts with documented design specifications:
* Preemption Law #1 (preemption policies are strong suggestions, not
guarantees): Opting out of preemption and originator status are intended as a
last resort when no other suitable allocations can be preempted.
* {*}{*}Quota Preemption Design Document: The specification states that
non-originators and pods with {{allowPreemption: true}} should receive higher
scores to prioritize regular allocations for preemption over originators and
opted-out pods.
h3. Test Verification
The following test demonstrates the ordering in
{{{}pkg/scheduler/objects/preemption_utilities_test.go{}}}:
{code:go}
func TestSortAllocationsBasedOnAsk_PreemptionOrdering(t *testing.T) {
node := NewNode(&si.NodeInfo{
NodeID: "node1",
SchedulableResource: &si.Resource{
Resources: map[string]*si.Quantity{"first": {Value:
100}},
},
})
res :=
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 10})
total :=
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 100})
ask :=
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 10})
regularPod := createAllocation("regularPod", "app1", node.NodeID, true,
false, 10, false, res)
originatorPod := createAllocation("originatorPod", "app1", node.NodeID,
true, true, 10, false, res)
optedOutPod := createAllocation("optedOutPod", "app1", node.NodeID,
false, false, 10, false, res)
allocations := []*Allocation{regularPod, originatorPod, optedOutPod}
SortAllocationsBasedOnAsk(allocations, total, ask)
// Current ordering:
// allocations[0]: optedOutPod
// allocations[1]: originatorPod
// allocations[2]: regularPod
//
// Expected ordering per design: regularPod should be placed at index 0
assert.Equal(t, allocations[0].GetAllocationKey(), "regularPod",
"Expected regularPod at index 0, but got: %s",
allocations[0].GetAllocationKey())
}
{code}
*Output:*
{code:java}
=== RUN TestSortAllocationsBasedOnAsk_PreemptionOrdering
preemption_utilities_test.go:320: Expected regularPod at index 0, but got:
optedOutPod
--- FAIL: TestSortAllocationsBasedOnAsk_PreemptionOrdering (0.00s)
{code}
h3. Impact
During Quota Preemption, reducing a queue's quota results in originator and
opted-out allocations being preempted before standard worker allocations.
h3. Proposed Solution
In {{{}pkg/scheduler/objects/preemption_utilities.go{}}}, assign the
higher-order score bits to unprotected allocations ({{{}allowPreemptSelf ==
true{}}} and {{{}!isOriginator{}}}):
{code:go}
var (
scoreAllowPreempt uint64 = 1 << 34
scoreNonOriginator uint64 = 1 << 33
)
func scoreAllocationBasedOnAsk(allocation *Allocation, ask *resources.Resource)
uint64 {
var score uint64 = 0
if !allocation.IsOriginator() {
score |= scoreNonOriginator
}
if allocation.IsAllowPreemptSelf() {
score |= scoreAllowPreempt
}
score += allocation.GetAllocatedResource().TypeMatching(ask)
return score
}
{code}
was:
h3. Overview
In {{{}pkg/scheduler/objects/preemption_utilities.go{}}},
{{SortAllocationsBasedOnAsk()}} and {{scoreAllocationBasedOnAsk()}} determine
victim ordering during preemption.
Under the current implementation, allocations with {{!allowPreemptSelf}}
(opted-out) or {{isOriginator}} (driver/manager) receive higher-order score
bits ({{{}scoreNoPreempt{}}} = {{{}1 << 34{}}}, {{scoreOriginator}} = {{{}1 <<
33{}}}):
{code:go}
func scoreAllocationBasedOnAsk(allocation *Allocation, ask *resources.Resource)
uint64 {
var score uint64 = 0
if allocation.IsOriginator() {
score |= scoreOriginator // 1 << 33
}
if !allocation.IsAllowPreemptSelf() {
score |= scoreNoPreempt // 1 << 34
}
score += allocation.GetAllocatedResource().TypeMatching(ask)
return score
}
{code}
Because {{SortAllocationsBasedOnAsk()}} orders allocations in descending order
({{{}scoreLeft > scoreRight{}}}), opted-out and originator allocations receive
higher scores and are placed at the beginning of the candidate slice rather
than at the end.
h3. Comparison with Design Documentation
This ordering conflicts with documented design specifications:
* {*}{{*}}Preemption Law #1 (preemption policies are strong suggestions, not
guarantees):{{*}}{*} Opting out of preemption and originator status are
intended as a last resort when no other suitable allocations can be preempted.
* {*}{{*}}Quota Preemption Design Document:{{*}}{*} The specification states
that non-originators and pods with {{allowPreemption: true}} should receive
higher scores to prioritize regular allocations for preemption over originators
and opted-out pods.
h3. Test Verification
The following test demonstrates the ordering in
{{{}pkg/scheduler/objects/preemption_utilities_test.go{}}}:
{code:go}
func TestSortAllocationsBasedOnAsk_PreemptionOrdering(t *testing.T) {
node := NewNode(&si.NodeInfo{
NodeID: "node1",
SchedulableResource: &si.Resource{
Resources: map[string]*si.Quantity{"first": {Value:
100}},
},
})
res :=
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 10})
total :=
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 100})
ask :=
resources.NewResourceFromMap(map[string]resources.Quantity{"first": 10})
regularPod := createAllocation("regularPod", "app1", node.NodeID, true,
false, 10, false, res)
originatorPod := createAllocation("originatorPod", "app1", node.NodeID,
true, true, 10, false, res)
optedOutPod := createAllocation("optedOutPod", "app1", node.NodeID,
false, false, 10, false, res)
allocations := []*Allocation{regularPod, originatorPod, optedOutPod}
SortAllocationsBasedOnAsk(allocations, total, ask)
// Current ordering:
// allocations[0]: optedOutPod
// allocations[1]: originatorPod
// allocations[2]: regularPod
//
// Expected ordering per design: regularPod should be placed at index 0
assert.Equal(t, allocations[0].GetAllocationKey(), "regularPod",
"Expected regularPod at index 0, but got: %s",
allocations[0].GetAllocationKey())
}
{code}
*Output:*
{code:java}
=== RUN TestSortAllocationsBasedOnAsk_PreemptionOrdering
preemption_utilities_test.go:320: Expected regularPod at index 0, but got:
optedOutPod
--- FAIL: TestSortAllocationsBasedOnAsk_PreemptionOrdering (0.00s)
{code}
h3. Impact
During Quota Preemption, reducing a queue's quota results in originator and
opted-out allocations being preempted before standard worker allocations.
h3. Proposed Solution
In {{{}pkg/scheduler/objects/preemption_utilities.go{}}}, assign the
higher-order score bits to unprotected allocations ({{{}allowPreemptSelf ==
true{}}} and {{{}!isOriginator{}}}):
{code:go}
var (
scoreAllowPreempt uint64 = 1 << 34
scoreNonOriginator uint64 = 1 << 33
)
func scoreAllocationBasedOnAsk(allocation *Allocation, ask *resources.Resource)
uint64 {
var score uint64 = 0
if !allocation.IsOriginator() {
score |= scoreNonOriginator
}
if allocation.IsAllowPreemptSelf() {
score |= scoreAllowPreempt
}
score += allocation.GetAllocatedResource().TypeMatching(ask)
return score
}
{code}
> SortAllocationsBasedOnAsk prioritizes originator and opted-out pods for
> preemption instead of protecting them
> -------------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3445
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3445
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Affects Versions: 1.9.0
> Reporter: weichen lai
> Assignee: weichen lai
> Priority: Major
>
> h3. Overview
> In {{{}pkg/scheduler/objects/preemption_utilities.go{}}},
> {{SortAllocationsBasedOnAsk()}} and {{scoreAllocationBasedOnAsk()}} determine
> victim ordering during preemption.
> Under the current implementation, allocations with {{!allowPreemptSelf}}
> (opted-out) or {{isOriginator}} (driver/manager) receive higher-order score
> bits ({{{}scoreNoPreempt{}}} = {{{}1 << 34{}}}, {{scoreOriginator}} = {{{}1
> << 33{}}}):
> {code:go}
> func scoreAllocationBasedOnAsk(allocation *Allocation, ask
> *resources.Resource) uint64 {
> var score uint64 = 0
> if allocation.IsOriginator() {
> score |= scoreOriginator // 1 << 33
> }
> if !allocation.IsAllowPreemptSelf() {
> score |= scoreNoPreempt // 1 << 34
> }
> score += allocation.GetAllocatedResource().TypeMatching(ask)
> return score
> }
> {code}
> Because {{SortAllocationsBasedOnAsk()}} orders allocations in descending
> order ({{{}scoreLeft > scoreRight{}}}), opted-out and originator allocations
> receive higher scores and are placed at the beginning of the candidate slice
> rather than at the end.
> h3. Comparison with Design Documentation
> This ordering conflicts with documented design specifications:
> * Preemption Law #1 (preemption policies are strong suggestions, not
> guarantees): Opting out of preemption and originator status are intended as a
> last resort when no other suitable allocations can be preempted.
> * {*}{*}Quota Preemption Design Document: The specification states that
> non-originators and pods with {{allowPreemption: true}} should receive higher
> scores to prioritize regular allocations for preemption over originators and
> opted-out pods.
> h3. Test Verification
> The following test demonstrates the ordering in
> {{{}pkg/scheduler/objects/preemption_utilities_test.go{}}}:
> {code:go}
> func TestSortAllocationsBasedOnAsk_PreemptionOrdering(t *testing.T) {
> node := NewNode(&si.NodeInfo{
> NodeID: "node1",
> SchedulableResource: &si.Resource{
> Resources: map[string]*si.Quantity{"first": {Value:
> 100}},
> },
> })
> res :=
> resources.NewResourceFromMap(map[string]resources.Quantity{"first": 10})
> total :=
> resources.NewResourceFromMap(map[string]resources.Quantity{"first": 100})
> ask :=
> resources.NewResourceFromMap(map[string]resources.Quantity{"first": 10})
> regularPod := createAllocation("regularPod", "app1", node.NodeID, true,
> false, 10, false, res)
> originatorPod := createAllocation("originatorPod", "app1", node.NodeID,
> true, true, 10, false, res)
> optedOutPod := createAllocation("optedOutPod", "app1", node.NodeID,
> false, false, 10, false, res)
> allocations := []*Allocation{regularPod, originatorPod, optedOutPod}
> SortAllocationsBasedOnAsk(allocations, total, ask)
> // Current ordering:
> // allocations[0]: optedOutPod
> // allocations[1]: originatorPod
> // allocations[2]: regularPod
> //
> // Expected ordering per design: regularPod should be placed at index 0
> assert.Equal(t, allocations[0].GetAllocationKey(), "regularPod",
> "Expected regularPod at index 0, but got: %s",
> allocations[0].GetAllocationKey())
> }
> {code}
> *Output:*
> {code:java}
> === RUN TestSortAllocationsBasedOnAsk_PreemptionOrdering
> preemption_utilities_test.go:320: Expected regularPod at index 0, but
> got: optedOutPod
> --- FAIL: TestSortAllocationsBasedOnAsk_PreemptionOrdering (0.00s)
> {code}
> h3. Impact
> During Quota Preemption, reducing a queue's quota results in originator and
> opted-out allocations being preempted before standard worker allocations.
> h3. Proposed Solution
> In {{{}pkg/scheduler/objects/preemption_utilities.go{}}}, assign the
> higher-order score bits to unprotected allocations ({{{}allowPreemptSelf ==
> true{}}} and {{{}!isOriginator{}}}):
> {code:go}
> var (
> scoreAllowPreempt uint64 = 1 << 34
> scoreNonOriginator uint64 = 1 << 33
> )
> func scoreAllocationBasedOnAsk(allocation *Allocation, ask
> *resources.Resource) uint64 {
> var score uint64 = 0
> if !allocation.IsOriginator() {
> score |= scoreNonOriginator
> }
> if allocation.IsAllowPreemptSelf() {
> score |= scoreAllowPreempt
> }
> score += allocation.GetAllocatedResource().TypeMatching(ask)
> return score
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
