Cool, thanks a ton.
Just one doubt. I have tried and have been using the join method that you
mentioned, in grafana for getting the hostnames as legends but how do I
extract the hostname from the expression in alertmanager? I mean we have
the expression but I am stuck at getting the nodename out of it.
expr: |
(node_filesystem_avail_bytes < 100000000 and
node_filesystem_size_bytes > 100000000)
* on (instance) group_left(nodename) node_uname_info
Now, how do I extract the nodename? $labels.nodename ?
On Thursday, March 5, 2020 at 9:43:33 PM UTC+5:30, Brian Candler wrote:
>
> Yes. There are two basic approaches.
>
> The first (and more complex) is to join node_uname_info to pick extra
> labels to add to your alert. Example (untested):
>
> - alert: DiskFull
> expr: |
> (node_filesystem_avail_bytes < 100000000 and
> node_filesystem_size_bytes > 100000000)
> * on (instance) group_left(nodename) node_uname_info
>
> More info:
> * https://www.robustperception.io/how-to-have-labels-for-machine-roles
> *
> https://www.robustperception.io/exposing-the-software-version-to-prometheus
> * https://www.robustperception.io/left-joins-in-promql
>
> The downside is that it makes all your alerting rules more complex.
>
> The second approach, specifically for your problem of having the hostnames
> in alerts, is to put a meaningful name in the "instance" label, rather than
> the IP address. This approach is described here:
>
> *
> https://www.robustperception.io/target-labels-are-for-life-not-just-for-christmas/
> *
> https://www.slideshare.net/roidelapluie/taking-advantage-of-prometheus-relabeling-109483749
>
> This makes not only alerts but also dashboards much more usable.
>
> Taking this to the limit, I create my targets files with entries of the
> form "foo/1.2.3.4" and use relabelling so that the name "foo" ends up in
> the instance label, and the IP address is the target which is actually
> scraped. Details:
> https://groups.google.com/d/msg/prometheus-users/VafEFQQKdWc/YrChiAePBQAJ
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/32b8eeea-ea0a-4c7f-901b-df2f05b81002%40googlegroups.com.
