> You have to scrape an artificial metric, and worry about details such as
how to get a warning if you forget one of the disk volumes in the threshold
metric.
You can also write the rule to use a default value if the threshold metric
doesn't exist. That's helpful if most of the drives use the same
threshold, and only a few need to be different. The article I linked to
shows how to do that.
> to add a label to just one metric ... do I have to resort to some
"metric_relabel_configs" magic?
Yes. You can add the label to all metrics from a given target by using the
target labels at scraping time. If you want to add that label to certain
timeseries only then you'll have to add them post-scraping using
metric_relabel_configs. You'd match the metric name and labels, and
conditionally add new labels.
I'm not going to write this for you, because:
(1) as I said, it's a bad way to build a monitoring system
(2) in the limit, you will end up with a separate rewriting rule for every
instance+volume combination
If you're going to do that, you may as well simply write separate alerting
rules for each volume:
expr:
windows_logical_disk_free_bytes{instance="PC1",volume="HarddiskVolume1"} /
windows_logical_disk_size_bytes < 0.5
expr:
windows_logical_disk_free_bytes{instance="PC1",volume="HarddiskVolume2"} /
windows_logical_disk_size_bytes < 0.1
expr:
windows_logical_disk_free_bytes{instance="PC1",volume="HarddiskVolume3"} /
windows_logical_disk_size_bytes < 0.05
... etc
This doesn't scale to thousands of alerting rules, but neither does metric
relabeling with thousands of rules.
On Sunday, 21 November 2021 at 22:03:36 UTC [email protected] wrote:
> > Just create a new timeseries called, say,
> "windows_logical_disk_used_threshold_percent".
> > Give it the same set of labels as "windows_logical_disk_free_bytes" - at
> least 'instance' and 'volume'.
>
> I understand now. The trick is that the labels match between the
> timeseries. Labels 'instance' and 'volume' would match between disk metric
> and threshold metric, so each disk gets mapped to its threshold.
>
> It looks workable, and is probably cleaner in a theoretical way. But it
> looks complicated. You have to learn about 'ignoring' and 'honor_labels'.
> You have to scrape an artificial metric, and worry about details such as
> how to get a warning if you forget one of the disk volumes in the threshold
> metric.
>
> I think that adding a label per disk metric is simpler, at least for a
> very small network like mine. I guess it would be very simple then to write
> an alert for disks where you forgot a threshold label.
>
> > [...]
> > Secondly, if you change the thresholds to use, the labelsets will change,
> > which means they become different timeseries.
>
> I don't think that becoming a different timeseries because a label has
> changed is really a problem in practice, at least for a simple monitoring
> system like mine.
>
> Anyway, even if it is not a good solution, I would still like to know the
> best way to add a label to just one metric inside a scraping target. That
> may be useful in other situations.
>
> Is there a nice way to do that? Or do I have to resort to some
> "metric_relabel_configs" magic?
>
> Best regards,
> rdiez
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/fc30115b-f3f9-4d13-a31f-3a6bae3d9323n%40googlegroups.com.
