Thanks for the tip! On Mon, Nov 1, 2021 at 4:47 PM Ben Kochie <[email protected]> wrote:
> I'd say no more than 100 per Prometheus server is a good rule of thumb. > One rule should cover every single instance of an alert. For example "CPU > Saturation" is one alert rule, You should create that rule and never have > it change. > > > https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/ > > On Mon, Nov 1, 2021 at 1:10 AM Jerry Lee <[email protected]> wrote: > >> I agree, we don’t do a good job governing the rule registration. Keeping >> no of alert rules less than a thousand could be a golden rule? >> >> On Mon, Nov 1, 2021 at 9:00 AM Ben Kochie <[email protected]> wrote: >> >>> If you have more than a couple hundred, you're probably doing something >>> wrong. Rules should be generic, one alert is designed to cover all >>> instances in a set. >>> >>> On Mon, Nov 1, 2021 at 12:54 AM Jerry Lee <[email protected]> wrote: >>> >>>> What's the guidance in terms of max number of alert rules in a single >>>> Promethues server? I guess it's up to the complexity of alert rules, volume >>>> of metric database and server capacity, but any experiences to deal with >>>> the high number of alert rules will be helpful. >>>> >>>> As of today, I have more than 15,000 rules and experiencing alert >>>> process delay around an hour. >>>> >>>> Thanks, >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/696e7c15-6bdb-410d-830f-1e61a45c689fn%40googlegroups.com >>>> <https://groups.google.com/d/msgid/prometheus-users/696e7c15-6bdb-410d-830f-1e61a45c689fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAP%3DM4SGicTkzLWuQMma0_T4fAqNA0Zxe2PTxE%3DB5XabqLAb93A%40mail.gmail.com.
