Thanks for the quick response Brian, I was checking same. We are not using
Value as label, only using in annotations. Below is the alert config.
Please have a look and suggest what changes can help us achieve better
alert notifications. One more help i would request is, like below alert
expression, we have alert rule for warning(which is if usage reaches more
than 80%), we get both Critical & warning alert notification if value goes
to more than 90%. Would really appreciate help.
- alert: OutOfDiskSpace-Crit
expr: ((node_filesystem_size_bytes{fstype!~"nfs.*"} -
node_filesystem_avail_bytes{fstype!~"nfs.*"}) /
node_filesystem_size_bytes{fstype!~"nfs.*"} * 100 > 90) * on (instance)
group_left(nodename) node_uname_info
for: 5m
labels:
severity: "CRITICAL"
annotations:
summary: "High Disk Usage on *{{ $labels.instance }}* - *{{
$labels.nodename }}* on the filesystem *{{ $labels.mountpoint }}*"
description: "Current Usage of {{ $labels.mountpoint }} = *{{ $value
| humanize }}%*"
identifier: "*Cluster:* `{{ $labels.cluster }}`, *node:* `{{
$labels.node }}`, *Mountpoint:* `{{ $labels.mountpoint }}` "
On Tuesday, March 10, 2020 at 11:00:30 PM UTC+5:30, Brian Brazil wrote:
>
> On Tue, 10 Mar 2020 at 17:10, Rahul Hada <[email protected] <javascript:>>
> wrote:
>
>> I have alerts configured for various metrics, and has set send_resolved :
>> true, now once alert is in active state, we are getting [Firing] emails as
>> configured, but we are getting the [resolved] email even if there is a
>> slight change in the values. For ex:- we get Firing alert when disk usage
>> is 98% and the [resolved] emails comes in if the values goes to 97% or
>> anything. Please suggest how to eradicate this issue.
>>
>
> Sounds like it might be
> https://www.robustperception.io/dont-put-the-value-in-alert-labels
>
>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> Brian Brazil
> www.robustperception.io
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7c692d96-f542-4c53-a5e2-61ee079d7588%40googlegroups.com.
