[
https://issues.apache.org/jira/browse/YUNIKORN-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YUNIKORN-331:
---------------------------------
Comment: was deleted
(was: hi [~adam.antal]
Sorry for not including enough info.
The problem today is we are publishing too many events to K8s.
If you look at the code:
https://github.com/apache/incubator-yunikorn-k8shim/blob/86cc199c00d44c1dde71c9f2faf5bc17ff28bbb7/pkg/plugin/predicates/predictor.go#L303-L304,
this is called in the core scheduling logic upon each allocation, which could
happen thousands of times per sec. For example, if a pod could not be allocated
onto any of the nodes due to some node taints, it runs a while and we will see
a huge number of dup events when we do "kubectl describe pod". So this gives us:
* good: we do not lose any of events, all pushed to K8s
* bad: overhead to the K8s event system (but gladly it aggregates the dup
events)
I think there are a few options we can evaluation:
# Shall we cache such events via the event cache system, and then push them in
1s interval just like what we have done for headRoom check?
# Add some rate-limit mechanism to reduce number of dup events
could you pls take a look and let me know your thought. thanks!
)
> Reduce the number events published to K8s event system when predicate fails
> ---------------------------------------------------------------------------
>
> Key: YUNIKORN-331
> URL: https://issues.apache.org/jira/browse/YUNIKORN-331
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - scheduler
> Reporter: Adam Antal
> Assignee: Adam Antal
> Priority: Major
>
> Currently the shim publishes pod events whenever there is a predicate
> evaluation failure. This is not efficient, we usually can see huge number of
> times that events got logged. We need to reduce the overhead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
