Yes, moving an expensive rule to a separate group can help. Each rule group is a separate goroutine, so you can distribute the work among many cores.
We have a couple of expensive rules that I had to hand-hack some sharding to reduce the eval time. Not a great solution, but it works. https://gitlab.com/gitlab-com/runbooks/-/blob/080c7bffab1051311f1c280f3deca3f6a7d5b934/rules/unicorn.yml#L104-111 On Sat, Feb 15, 2020 at 4:11 PM John Bryan Sazon <[email protected]> wrote: > Thanks! I saw something with this query: > > prometheus_rule_group_last_duration_seconds > > prometheus_rule_group_interval_seconds > > I had issues with recording rules error previously and I had to increase > the following default values to solve those issues. > > - --query.lookback-delta=6m # default is 5m > - --query.max-samples=100000000 # default is 50000000 > - --query.timeout=4m # default is 2m > > If I move the long duration recording rule to a specific new group, will > that help? > > On Saturday, 15 February 2020 16:02:49 UTC+1, Julien Pivotto wrote: >> >> On 15 Feb 06:52, John Bryan Sazon wrote: >> > I've been seeing this metric >> > (prometheus_rule_group_iterations_missed_total) increment all the time. >> In >> > the rules UI, I don't see any errors. Every rule is *OK*. I also don't >> see >> > anything unusual from the logs. >> >> I means that the rules take too long to be evaluated or that your >> prometheus server is overloaded. >> >> You can use prometheus_rule_group_last_duration_seconds > >> prometheus_rule_group_interval_seconds to see if you have an issue with >> the duration of the group evaluation. >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/d7181460-0513-438c-80ba-e3245820c71f%40googlegroups.com. >> >> >> >> -- >> (o- Julien Pivotto >> //\ Open-Source Consultant >> V_/_ Inuits - https://www.inuits.eu >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/0993ee96-f910-43f4-8489-9731da202767%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/0993ee96-f910-43f4-8489-9731da202767%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmo%3DgFu%2BzQT%2BQE0vhdcHoGnL5TTHrJrRrVtN9nn7wYN6_A%40mail.gmail.com.
