Yes. There are two basic approaches.
The first (and more complex) is to join node_uname_info to pick extra
labels to add to your alert. Example (untested):
- alert: DiskFull
expr: |
(node_filesystem_avail_bytes < 100000000 and
node_filesystem_size_bytes > 100000000)
* on (instance) group_left(nodename) node_uname_info
More info:
* https://www.robustperception.io/how-to-have-labels-for-machine-roles
*
https://www.robustperception.io/exposing-the-software-version-to-prometheus
* https://www.robustperception.io/left-joins-in-promql
The downside is that it makes all your alerting rules more complex.
The second approach, specifically for your problem of having the hostnames
in alerts, is to put a meaningful name in the "instance" label, rather than
the IP address. This approach is described here:
*
https://www.robustperception.io/target-labels-are-for-life-not-just-for-christmas/
*
https://www.slideshare.net/roidelapluie/taking-advantage-of-prometheus-relabeling-109483749
This makes not only alerts but also dashboards much more usable.
Taking this to the limit, I create my targets files with entries of the
form "foo/1.2.3.4" and use relabelling so that the name "foo" ends up in
the instance label, and the IP address is the target which is actually
scraped. Details:
https://groups.google.com/d/msg/prometheus-users/VafEFQQKdWc/YrChiAePBQAJ
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/4add91a9-dc80-4581-bd84-f002f714c19e%40googlegroups.com.
