Victor Zhou created YUNIKORN-3139:
-------------------------------------
Summary: Preemption throughput constrained by incorrect counter
decrement logic
Key: YUNIKORN-3139
URL: https://issues.apache.org/jira/browse/YUNIKORN-3139
Project: Apache YuniKorn
Issue Type: Bug
Reporter: Victor Zhou
h2. Summary
The preemption attempt counter (`preemptAttemptsRemaining`) is decremented
regardless of whether the preemption attempt succeeds or not. When a queue has
many allocation asks requiring preemption, the scheduler frequently encounters
asks that have already triggered preemption in previous cycles. These duplicate
attempts fail precondition checks due to:
1. Frequency throttling: 15-second minimum interval between preemption attempts
(`preemptAttemptFrequency`)
2. Already triggered: The ask has previously triggered preemption
(`HasTriggeredPreemption()` returns true)
Since the counter is decremented before these precondition checks, it gets
wasted on asks that cannot actually trigger preemption. This severely limits
preemption throughput for large-scale applications, causing multi-hour delays
for applications with 1000+ pods.
h2. How to reproduce
Environment
- Cluster at near-full capacity with many low-priority batch jobs
- High-priority Spark job submitted: 1 driver + 1000+ executors
- Sufficient preemptable resources available
Expected Behavior:
- Executors should trigger preemption of low-priority jobs
- Full allocation within 25-30 minutes
Actual Behavior:
- Executors took over 2 hour to allocate
- Preemption proceeded extremely slowly despite available victims
Potential Root Cause
Preemption throughput bottleneck:
- maxPreemptionsPerQueue = 10 attempts per scheduling cycle
- Counter decremented on precondition failures (no actual preemption attempted)
https://github.com/apache/yunikorn-core/blob/master/pkg/scheduler/objects/application.go#L1017
h2. Proposed solution
Only decrement the counter when a preemption attempt actually occurs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
