On 02/02/2021 18:30, [email protected] wrote:
Hi. I am using Prometheus version 2.22.1 and I am using custom
thresholds for many of my alerts using the recording rules. But for
Disk Space Utilization I am getting stuck where I need to set a custom
threshold only for a single mount point of a particular IP. For eg. My
default disk space utilization threshold is 90%. On the IP x.x.x.x,
there are 2 mounts - vol1 and vol2. I want to set the threshold for
vol2 as 95%. To set the threshold to 95% for all the mount points of
an IP is easy and can be done in the way I have done below but How do
I approach setting the custom threshold for one of the mount point?
Sample of custom thresholding that I am doing for one of the other
alerts - Number of Threads.
- record: custom_critical
expr: (up{job=~"node.*",instance="x.x.x.x:9100"}) + 9999
Alert Expression:
((node_processes_threads) > on(instance) group_left() (custom_critical
or on(instance) count by (instance)(node_processes_threads) * 0 + 4000))
A common way is to use timeseries to set the thresholds:
https://www.robustperception.io/using-time-series-as-alert-thresholds
Also, rather than simple % thresholds the predict_linear function can be
very useful - alerting instead when space is predicted to be exhausted
within a particular time period. See
https://www.robustperception.io/tag/predict_linear
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/c1f365b7-dd22-b226-2554-b27a5b142e83%40Jahingo.com.
