[
https://issues.apache.org/jira/browse/YUNIKORN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikandan R reassigned YUNIKORN-3139:
--------------------------------------
Assignee: Manikandan R
> Preemption throughput constrained by incorrect counter decrement logic
> ----------------------------------------------------------------------
>
> Key: YUNIKORN-3139
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3139
> Project: Apache YuniKorn
> Issue Type: Bug
> Reporter: Victor Zhou
> Assignee: Manikandan R
> Priority: Critical
>
> h2. Summary
> The preemption attempt counter (`preemptAttemptsRemaining`) is decremented
> regardless of whether the preemption attempt succeeds or not. When a queue
> has many allocation asks requiring preemption, the scheduler frequently
> encounters asks that have already triggered preemption in previous cycles.
> These duplicate attempts fail precondition checks due to:
> 1. Frequency throttling: 15-second minimum interval between preemption
> attempts (`preemptAttemptFrequency`)
> 2. Already triggered: The ask has previously triggered preemption
> (`HasTriggeredPreemption()` returns true)
> Since the counter is decremented before these precondition checks, it gets
> wasted on asks that cannot actually trigger preemption. This severely limits
> preemption throughput for large-scale applications, causing multi-hour delays
> for applications with 1000+ pods.
> h2. How to reproduce
> Environment
> - Cluster at near-full capacity with many low-priority batch jobs
> - High-priority Spark job submitted: 1 driver + 1000+ executors
> - Sufficient preemptable resources available
> Expected Behavior:
> - Executors should trigger preemption of low-priority jobs
> - Full allocation within 25-30 minutes
> Actual Behavior:
> - Executors took over 2 hour to allocate
> - Preemption proceeded extremely slowly despite available victims
>
> Potential Root Cause
> Preemption throughput bottleneck:
> - maxPreemptionsPerQueue = 10 attempts per scheduling cycle
> - Counter decremented on precondition failures (no actual preemption
> attempted)
> https://github.com/apache/yunikorn-core/blob/master/pkg/scheduler/objects/application.go#L1017
> h2. Proposed solution
> Only decrement the counter when a preemption attempt actually occurs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
