[
https://issues.apache.org/jira/browse/YUNIKORN-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YUNIKORN-333:
---------------------------------
Description:
The problem today is we are publishing too many events to K8s.
If you look at the code:
https://github.com/apache/incubator-yunikorn-k8shim/blob/86cc199c00d44c1dde71c9f2faf5bc17ff28bbb7/pkg/plugin/predicates/predictor.go#L303-L304,
this is called in the core scheduling logic upon each allocation, which could
happen thousands of times per sec. For example, if a pod could not be allocated
onto any of the nodes due to some node taints, it runs a while and we will see
a huge number of dup events when we do "kubectl describe pod". So this gives us:
- good: we do not lose any of events, all pushed to K8s
- bad: overhead to the K8s event system (but gladly it aggregates the dup
events)
I think there are a few options we can evaluation:
- Shall we cache such events via the event cache system, and then push them in
1s interval just like what we have done for headRoom check?
- Add some rate-limit mechanism to reduce number of dup events
could you pls take a look and let me know your thought. thanks!
was:
Currently the shim publishes pod events whenever there is a predicate
evaluation failure. This is not efficient, we usually can see huge number of
times that events got logged. We need to reduce the overhead.
> Reduce the number events published to K8s event system when predicate fails
> ---------------------------------------------------------------------------
>
> Key: YUNIKORN-333
> URL: https://issues.apache.org/jira/browse/YUNIKORN-333
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - scheduler
> Reporter: Adam Antal
> Assignee: Adam Antal
> Priority: Major
>
> The problem today is we are publishing too many events to K8s.
> If you look at the code:
> https://github.com/apache/incubator-yunikorn-k8shim/blob/86cc199c00d44c1dde71c9f2faf5bc17ff28bbb7/pkg/plugin/predicates/predictor.go#L303-L304,
> this is called in the core scheduling logic upon each allocation, which
> could happen thousands of times per sec. For example, if a pod could not be
> allocated onto any of the nodes due to some node taints, it runs a while and
> we will see a huge number of dup events when we do "kubectl describe pod". So
> this gives us:
> - good: we do not lose any of events, all pushed to K8s
> - bad: overhead to the K8s event system (but gladly it aggregates the dup
> events)
> I think there are a few options we can evaluation:
> - Shall we cache such events via the event cache system, and then push them
> in 1s interval just like what we have done for headRoom check?
> - Add some rate-limit mechanism to reduce number of dup events
> could you pls take a look and let me know your thought. thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
