You could do that - for example, in your inventory system, you expose the
status of the targets as a prometheus exporter endpoint:
monitoring_active{instance="foo"} 1
monitoring_active{instance="bar"} 0
Then you scrape this to create a new set of timeseries, and you write your
alerting expressions to include this values when deciding whether to alert
or not. These rules become a bit awkward, using one-to-one or many-to-one
vector matches.
But personally, I would do it the other way around. What I suggest is that
you generate the list of targets to scrape *from* the inventory system in
the first place. Then you can either:
1. Not include machines with monitoring=no in the list of targets (so they
don't get scraped at all); or
2. Scrape all targets, but add monitoring="no" or monitoring="yes" as a
target label. You can then use that label in your alerting rules:
# old
expr: up == 0
# new
expr: up{monitoring="yes"} == 0
Or even simpler, you can use this target label in your alertmanager rules
to route the alert to a null endpoint. That is, you still generate alerts
for all targets, you just don't deliver them if they are labeled with
monitoring="no".
To do this, you will use the "service discovery" features of prometheus, so
that your monitoring system *pushes* information to prometheus telling it
what to scrape and how. The simplest mechanism is the "file_sd" mechanism,
where you just write the targets into a file.
This is what I do:
- my inventory is in Netbox
- I wrote some code <https://github.com/candlerb/netbox-prometheus> to read
the inventory via Netbox API every 5 minutes and write out the prometheus
target files (for file_sd_targets)
An example of the generated targets file:
# Auto-generated from Netbox, do not edit as your changes will be
overwritten!
- labels:
netbox_type: device
targets:
- nuc1 10.12.255.11
- nuc2 10.12.255.12
- storage1 10.12.255.5
You could generate two sets of targets, one with label "monitoring: yes"
and one with label "monitoring: no"
This has the very nice side benefit that adding a device to your inventory
will *automatically* add it to prometheus monitoring.
HTH,
Brian.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8327d37c-bcf8-4e95-8406-0b680d71e15dn%40googlegroups.com.
