Generate additional static timeseries such as:
es_exporter_percentiles_values_95_0_threshold{httppath="path1",app="app1"}
10
es_exporter_percentiles_values_95_0_threshold{httppath="path2",app="app2"} 5
es_exporter_percentiles_values_95_0_threshold{httppath="path3",app="app3"} 3
Then your alerting rule becomes just:
es_exporter_percentiles_values_95_0 > on(httppath,app)
es_exporter_percentiles_values_95_0_threshold
Those static timeseries can be put onto a webserver that you scrape, or
using node_exporter textfile_collector. You may need to add extra labels
to your threshold timeseries and on(...) clause if there is overlap, e.g.
between different environments, to ensure there's always a 1:1 match
between value and threshold.
Original idea here:
https://www.robustperception.io/using-time-series-as-alert-thresholds
which also shows how you can have a default threshold for those which
aren't explicitly set.
On Wednesday, 24 November 2021 at 10:58:21 UTC [email protected] wrote:
> Hi Team,
>
> Any suggestion is very much appreciated.
>
> I have an alert for breach of 95 percentile threshold on API requests. Now
> problem is, different API requests has different threshold, so I cannot
> hard code the alerts as I would have to add 10 alerts for 10 different APIs
> with their thresholds.
>
> The alerts would like something like this
> es_exporter_percentiles_values_95_0{httppath="path1",app="app1"} > 10
> es_exporter_percentiles_values_95_0{httppath="path2",app="app2"} > 5
> es_exporter_percentiles_values_95_0{httppath="path3",app="app3"} > 3
>
> Doesn't want to use or as again wanted to do this in a better smarter way.
> Tried using recording rules, but that would result in adding recording
> rules for 20 apps * 4 env * 5 http paths.
>
> Any better way, so that I have only one alert definition but can be
> applied against all apps, paths and envs.
>
> Thanks
> Eswar
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/b52f7b3d-5f13-4a3b-8177-3303ed6d112fn%40googlegroups.com.
