Kishan Ghetia created YUNIKORN-3406:
---------------------------------------
Summary: Support queue-headroom-based limits for application
unschedulable backoff
Key: YUNIKORN-3406
URL: https://issues.apache.org/jira/browse/YUNIKORN-3406
Project: Apache YuniKorn
Issue Type: Improvement
Components: core - scheduler
Reporter: Kishan Ghetia
Application.tryAllocate() currently stops after a fixed number of unschedulable
asks. For example with config:
application.unschedasks.backoff: "50"
application.unschedasks.backoff.delay: "5s"
With large Spark executor bursts, the queue may have quota for more than 50
asks, but later asks are not marked {{SchedulingAttempted}} before the
application backs off. Consequently, the outstanding-request scan cannot mark
those pods {{{}Unschedulable{}}}, delaying autoscaling even though queue quota
is available.
Proposal: add an option for {{tryAllocate()}} to use the remaining queue
quota/headroom as the unschedulable-ask limit. It should attempt enough sorted
requests to fill the available queue quota, using the same effective limit as
outstanding-request marking, and then apply the existing application backoff.
This preserves the backoff’s protection against unbounded node and predicate
checks while exposing all autoscaling demand that the queue can currently
accept.
Acceptance criteria:
* When queue headroom supports more than the fixed count, enough asks are
attempted to fill that headroom.
* Asks beyond available queue headroom are not exposed for autoscaling.
* Existing backoff behavior remains after the quota-based limit is reached.
* Tests cover more than 50 asks and mixed request sizes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
