I'd say no more than 100 per Prometheus server is a good rule of thumb. One rule should cover every single instance of an alert. For example "CPU Saturation" is one alert rule, You should create that rule and never have it change.
https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/ On Mon, Nov 1, 2021 at 1:10 AM Jerry Lee <[email protected]> wrote: > I agree, we don’t do a good job governing the rule registration. Keeping > no of alert rules less than a thousand could be a golden rule? > > On Mon, Nov 1, 2021 at 9:00 AM Ben Kochie <[email protected]> wrote: > >> If you have more than a couple hundred, you're probably doing something >> wrong. Rules should be generic, one alert is designed to cover all >> instances in a set. >> >> On Mon, Nov 1, 2021 at 12:54 AM Jerry Lee <[email protected]> wrote: >> >>> What's the guidance in terms of max number of alert rules in a single >>> Promethues server? I guess it's up to the complexity of alert rules, volume >>> of metric database and server capacity, but any experiences to deal with >>> the high number of alert rules will be helpful. >>> >>> As of today, I have more than 15,000 rules and experiencing alert >>> process delay around an hour. >>> >>> Thanks, >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/696e7c15-6bdb-410d-830f-1e61a45c689fn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/696e7c15-6bdb-410d-830f-1e61a45c689fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqR6bH7rS8rwNYmD12EtnPm7zqDWAqt60Fnn2pHzXtG1Q%40mail.gmail.com.
