On Tue, 10 Mar 2020 at 17:41, Rahul Hada <[email protected]> wrote: > Thanks for the quick response Brian, I was checking same. We are not using > Value as label, only using in annotations. Below is the alert config. > Please have a look and suggest what changes can help us achieve better > alert notifications. One more help i would request is, like below alert > expression, we have alert rule for warning(which is if usage reaches more > than 80%), we get both Critical & warning alert notification if value goes > to more than 90%. Would really appreciate help. >
That looks okay to me, though watch for https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 Brian > > - alert: OutOfDiskSpace-Crit > expr: ((node_filesystem_size_bytes{fstype!~"nfs.*"} - > node_filesystem_avail_bytes{fstype!~"nfs.*"}) / > node_filesystem_size_bytes{fstype!~"nfs.*"} * 100 > 90) * on (instance) > group_left(nodename) node_uname_info > for: 5m > labels: > severity: "CRITICAL" > annotations: > summary: "High Disk Usage on *{{ $labels.instance }}* - *{{ > $labels.nodename }}* on the filesystem *{{ $labels.mountpoint }}*" > description: "Current Usage of {{ $labels.mountpoint }} = *{{ $value > | humanize }}%*" > identifier: "*Cluster:* `{{ $labels.cluster }}`, *node:* `{{ > $labels.node }}`, *Mountpoint:* `{{ $labels.mountpoint }}` " > > > On Tuesday, March 10, 2020 at 11:00:30 PM UTC+5:30, Brian Brazil wrote: >> >> On Tue, 10 Mar 2020 at 17:10, Rahul Hada <[email protected]> wrote: >> >>> I have alerts configured for various metrics, and has set send_resolved >>> : true, now once alert is in active state, we are getting [Firing] emails >>> as configured, but we are getting the [resolved] email even if there is a >>> slight change in the values. For ex:- we get Firing alert when disk usage >>> is 98% and the [resolved] emails comes in if the values goes to 97% or >>> anything. Please suggest how to eradicate this issue. >>> >> >> Sounds like it might be >> https://www.robustperception.io/dont-put-the-value-in-alert-labels >> >> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Brian Brazil >> www.robustperception.io >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/7c692d96-f542-4c53-a5e2-61ee079d7588%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/7c692d96-f542-4c53-a5e2-61ee079d7588%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLqgRJfOLci1qzSY4JkAvhT1fQc%2BuOaRudFWpUL-nfe6kw%40mail.gmail.com.
