On Wed, 5 Aug 2020 at 09:56, Federico Buti <[email protected]> wrote:
> Hi. > > Thanks for the reply Brian. > So one should not alert on absence of a metric? Never ever? Just on the > upness of the targets? > Generally you should alert on the absence of up, as that indicates something has either gone wrong with service discovery or the service has disappeared and no longer exists. Alerting on absence of scraped metrics only really applies in niche cases where a target has a bug that'd result in it sometimes not exposing metrics. The blackbox exporter doesn't have such a bug that I'm aware of. Brian > > --- > Federico Buti > > > On Wed, 5 Aug 2020 at 10:46, Brian Brazil < > [email protected]> wrote: > >> On Wed, 5 Aug 2020 at 09:32, Federico Buti <[email protected]> wrote: >> >>> Hi all. >>> >>> A few months ago we introduced target down rules to keep track of >>> targets that were missing. The rules are relatively simple being something >>> like e.g. >>> >>> alert: target_down_slower_scraping_jobs >>> expr: up{job=~"monitoring-scripts-5m|monitoring-scripts-hourly"} == 0 >>> for: 13m >>> labels: >>> severity: average >>> annotations: >>> // annotations here >>> >>> >>> >> >> >>> A few days ago we wanted to introduce absence rules and we added them >>> for both targets and metrics. That is all ok but with a side effect that we >>> didn't consider, i.e. a metrics absent alert would of course spawn if the >>> corresponding target is down. >>> >> >> That doesn't sound right, make sure your absent rules were on "up". >> >> Brian >> >> >>> Looking into it I've found this blog post >>> <https://www.robustperception.io/absent-alerting-for-scraped-metrics> >>> proposing >>> to use unless binary operator but I'm not sure I've understood the usage >>> and its implications. >>> >>> Unless returns the first metric unless we have some match for the >>> second. If I write something like >>> >>> expr: up{job="node"} == 1 unless absent(check_success{check="xxxxx", >>> stack="yyy",environment="zzz"}) >>> >>> I'm just going to return the upness if everything is fine with the node. >>> Isn't that wrong? I mean, that would result in an alert because the node is >>> up, which is not what we want. Even changing that to 0 would not solve the >>> problem since we would still return the absence. Maybe changing to zero and >>> inverting the two? But then wouldn't I have duplicated alerts for the >>> upness? >>> >>> Is there a way to make sure absent rules take in account targets down? >>> Or should I approach the issue in some other different ways which I'm not >>> considering now? >>> >>> Thanks in advance, >>> F. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/7758b7cc-79e9-4b0a-b39f-bff6bcb62d4co%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/7758b7cc-79e9-4b0a-b39f-bff6bcb62d4co%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Brian Brazil >> www.robustperception.io >> > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLqa4G7x%3DECYCgrTzpju5NdBUW%3DFgdyrP7b2NEGqAZDJVw%40mail.gmail.com.
