> I am encountering challenges with configuring Prometheus and Alertmanager
> for my application's alarm system. Below are the configurations I am
> currently using:
>
> *prometheus.yml:*
>
> Scrape Interval: 1h
This scrape interval is far too high. Although it's not well documented,
you can't set scrape_interval higher than two or three minutes without
causing seriously weird issues, where your rules may not see metrics
because Prometheus considers the metrics stale. Prometheus considers
metrics stale if the most recent sample is more than five minutes old;
this time is not adjustable as far as I know. I believe you've already
seen signs of this from your other problems, but really, as far as I
know such a configuration basically isn't supported.
(In my view this is such a problem that Prometheus should at least
require a forced 'I know what I'm doing, really I want this' command
line option to accept a scrape interval that's larger than the staleness
interval, or maybe even within ten seconds or so of it.)
I believe that all rule evaluation intervals similarly need to be no
more than five minutes because of the stale metrics issue, since both
recording rules and alerting rules generate metrics (the recording rules
generate their metrics in an obvious way, the alerting rules generate
ALERTS metrics and some other ones). It's possible that alerts don't go
stale inside Prometheus despite their metrics going stale, but I
wouldn't count on this.
(Although it's possible that metrics from recording and/or alert rules
are special and are exempted from staleness, I would be surprised.)
Prometheus scrapes different targets at different offsets within their
scrape interval, so you can't synchronize scrapes and rule evaluations
the way you apparently want to. The time offset for any particular
scrape target is deterministic but not predictable (and it may change
between eg Prometheus releases, or even on a Prometheus restart).
Prometheus does this to spread out the load of scraping more or less
evenly across the scrape interval, rather than descending on all targets
simultaneously every X seconds or minutes.
I believe that recording rules and alerting rules similarly may have
their evaluation time happen at different offsets within their
evaluation interval. This is done for the similar reason of spreading
out the internal load of rule evaluations across time.
- cks
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/1394369.1711750147%40apps0.cs.toronto.edu.
