I observe that unless blackbox_exporter itself fails you should never see
up{job="blackbox"} == 0; a real alert is from probe_success == 0.
Hence I suggest separate expressions:
expr: up{job!="blackbox"} == 0 # other targets are down
expr: probe_success == 0 # blackbox target has failed
expr: min by (job) (up{job="blackbox"}) == 0 # a single alert, for
"blackbox_exporter itself has failed" (which alerts if *any* scrape to
blackbox fails)
I think this is cleaner than inhibits. Using inhibits you'd still need an
expression like max(up{job="blackbox"}) == 0 to be used to inhibit all the
other blackbox_exporter up==0 alerts.
Or, you could have a routing rule which matches job="blackbox", and under
that have group_by: ['job']. Then you'll receive one message, with up to
50 separate alerts listed within it. However, it will also group together
all other blackbox alerts like probe_success == 0, unless you add a
specific label and test for that in your routing rule.
On Tuesday, 1 August 2023 at 15:48:16 UTC+1 Александр Админ wrote:
> Hello,
> we have a blackbox exporter which is probing around 50 targets.
> We also have an alert rule: *up == 0 *
> so we can know then certain targets are not scraped.
>
> *The problem is*: if blackbox exporter goes down, we receive 50 alerts.
> One for each "down" target (instance). We would like to receive only single
> alert.
> I was thinking that inhibit_rule might solve that problem.
> But it's not.
> Here is the config:
> *inhibit_rules: *
> * - equal: *
> * - instance *
> * source_matchers: *
> * - alertname="Prometheus target state" *
> * target_matchers: *
> * - alertname="Prometheus target state" *
> * - job="Blackbox exporter job"*
>
> Does anyone know how we can solve that problem? Thanks!
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/560f9755-9388-4b0e-b5c2-5129e7891cb7n%40googlegroups.com.
