> Each rule group is a separate goroutine, so you can distribute the work among many cores.
Thanks for that useful information! On Saturday, 15 February 2020 16:26:32 UTC+1, Ben Kochie wrote: > > Yes, moving an expensive rule to a separate group can help. Each rule > group is a separate goroutine, so you can distribute the work among many > cores. > > We have a couple of expensive rules that I had to hand-hack some sharding > to reduce the eval time. Not a great solution, but it works. > > > https://gitlab.com/gitlab-com/runbooks/-/blob/080c7bffab1051311f1c280f3deca3f6a7d5b934/rules/unicorn.yml#L104-111 > > On Sat, Feb 15, 2020 at 4:11 PM John Bryan Sazon <[email protected] > <javascript:>> wrote: > >> Thanks! I saw something with this query: >> >> prometheus_rule_group_last_duration_seconds > >> prometheus_rule_group_interval_seconds >> >> I had issues with recording rules error previously and I had to increase >> the following default values to solve those issues. >> >> - --query.lookback-delta=6m # default is 5m >> - --query.max-samples=100000000 # default is 50000000 >> - --query.timeout=4m # default is 2m >> >> If I move the long duration recording rule to a specific new group, will >> that help? >> >> On Saturday, 15 February 2020 16:02:49 UTC+1, Julien Pivotto wrote: >>> >>> On 15 Feb 06:52, John Bryan Sazon wrote: >>> > I've been seeing this metric >>> > (prometheus_rule_group_iterations_missed_total) increment all the >>> time. In >>> > the rules UI, I don't see any errors. Every rule is *OK*. I also don't >>> see >>> > anything unusual from the logs. >>> >>> I means that the rules take too long to be evaluated or that your >>> prometheus server is overloaded. >>> >>> You can use prometheus_rule_group_last_duration_seconds > >>> prometheus_rule_group_interval_seconds to see if you have an issue with >>> the duration of the group evaluation. >>> >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/d7181460-0513-438c-80ba-e3245820c71f%40googlegroups.com. >>> >>> >>> >>> >>> -- >>> (o- Julien Pivotto >>> //\ Open-Source Consultant >>> V_/_ Inuits - https://www.inuits.eu >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/0993ee96-f910-43f4-8489-9731da202767%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/0993ee96-f910-43f4-8489-9731da202767%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f65c32cf-3387-4900-bbe6-0bc750d6c564%40googlegroups.com.
