On Sunday, 21 November 2021 at 20:20:24 UTC [email protected] wrote: > First of all, thanks for your answers. > > > https://www.robustperception.io/using-time-series-as-alert-thresholds > > I do not understand how that article can help me. If I understood that > correctly, it seems to describe a way for different teams to define their > own thresholds. >
It shows the general idea of using one timeseries as the threshold to another. You can forget the "teams" idea entirely. Just create a new timeseries called, say, "windows_logical_disk_used_threshold_percent". Give it the same set of labels as "windows_logical_disk_free_bytes" - at least 'instance' and 'volume'. Then alert on: 100.0 - 100 * (windows_logical_disk_free_bytes / windows_logical_disk_size_bytes) > windows_logical_disk_used_threshold_percent That's it. If the label sets are not exactly the same, in particular if windows_logical_disk_free_bytes has some extra labels which you want to ignore, then using ignoring(...) <https://prometheus.io/docs/prometheus/latest/querying/operators/#one-to-one-vector-matches> on the extra labels. You can generate the static timeseries by using node_exporter textfile collector, or by putting up a static web page somewhere containing all the threshold metrics and scraping it. Either way you'll be overriding the "instance" label, so you'll need the "honor_labels <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config>" setting in the scrape job, or else some relabelling. > At that point I thought I could add the corresponding threshold label to > each separate disk, but that's what I am struggling to do in Prometheus. Do > I have to resort to some "metric_relabel_configs" magic? Could you provide > an example on how to do that? > > I think that's a poor way to do it. Firstly, you're hard-coding information on *how to alert* into the metrics themselves. The metrics should purely represent the information collected, since they may be used by different systems for different purposes. Secondly, if you change the thresholds to use, the labelsets will change, which means they become different timeseries. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3d7f22aa-279c-4138-997c-7a6c2e0670ccn%40googlegroups.com.
