[
https://issues.apache.org/jira/browse/YUNIKORN-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17791830#comment-17791830
]
Wilfred Spiegelenburg commented on YUNIKORN-2212:
-------------------------------------------------
Unschedulable is managed by the quotas. Setting the fact that the scheduler
attempted to schedule the request or not will not slow things down. If the
quotas on the queues are high and you have a high demand pending the check will
take time to build up the list of all request that should be marked.
YuniKorn schedules eagerly. If the first pending request does not fit we will
check the next. That will cause all pending requests to quickly be tagged as
attempted and we are still in the same situation.
We need to become way smarter when looking at triggering the unschedulable
change. For instance why are we triggering any of this when the root queue
usage percentage is really low? I could have a cluster with 1000 nodes, the
cluster is only using 1 node out of the 1000. We should not trigger anything at
that point.
Second point: In a on-premise cluster we do not want/need to mark the pods for
autoscaling. We should be able to turn this off completely if not needed.
We also do not keep state for all of the asks that have triggered upscaling. If
we have checked the ask and marked it for autoscaling why do we do it again,
and again? That would also take into account that autoscaling does still take
~30 seconds in a cluster. It is marked and thus done.
I do not think your changes in the PR go far enough [~pbacsko] If we are making
changes we need to solve more than just first performance piece and fix/enhance
behaviour. In YuniKorn-277 I did a small write up on some of the limitations
and issues with the current way of auto scaling triggering.
> Request should not be outstanding if scheduling has not been attempted
> ----------------------------------------------------------------------
>
> Key: YUNIKORN-2212
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2212
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Priority: Major
> Labels: pull-request-available
> Attachments: outstandingRequests-1.png
>
>
> In every second, we collect "outstanding requests", ie those which cannot be
> scheduled.
> Problem is, the scheduling cycle might not even tried to schedule those pods.
> If that's the case, we mistakenly set them to "Unschedulable", which can
> trigger autoscaling if the cluster autoscaler happens to run at the beginning
> of the next scan interval.
> Another thing to consider is when we need to mark them as Unschedulable. Eg.
> tryPreemption() succeeded, do we still need new nodes? This can addressed in
> a separate JIRA.
> This issue also shows up during performance testing. Since we submit a lot of
> pods to Yunikorn, {{Scheduler.inspectOutstandingRequests()}} finds&collects
> them and subsequently generates a lot of API server updates. This is a
> special edge case, but on busy clusters, something similar can happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
