This might be an XY problem, because it is often better to have a defined
"up/down" metric (with value 1/0), which tells you whether something worked
or not, rather than alerting on presence or absence of a metric.
However, to answer your question directly, I think you would need to
include some condition saying whether that metric *should* be there or not
- which is the presence of some other metric. The "up" metric added by all
scrape jobs can be useful for this. In this case, I expect
*up{job="jenkins"}* will exist, if and only if you have a 'jenkins' scrape
job in that region. Therefore maybe something like this will do what you
want:
absent(jenkins_up{job="jenkins"}) unless on (job) absent(up{job="jenkins"})
which I think may simplify, if the 'jenkins_up' metric is only scraped by
the 'jenkins' job, to this (not sure):
absent(jenkins_up) unless on () absent(up{job="jenkins"})
On Sunday, 3 October 2021 at 19:40:23 UTC+1 Dan S wrote:
> Hi,
>
> Looking for some general advice about shared prom alert rules between
> regions. We currently push the same alert rules to all regions, and
> sometimes we run into situations where we have a specific job in region X
> but not Y.
>
> This is fine for basic cases, such as *up{job="jenkins"} == 0* which will
> be ignored in regions where there's no jenkins job present (or could easily
> specify region="X").
>
> But in some situations I'd like to use absent on a metric that often has
> gaps for example
> *absent(jenkins_up{job="jenkins"})*
> This would trigger in all regions, whether or not there's a job "jenkins"
> (obviously because it's triggering on the missing metrics) even if I try to
> be more specific: *absent(jenkins_up{job="jenkins", region="US"}).*
>
> Any suggestions how I can craft an alert query using absent() in on
> metrics that don't appear in all regions? So that if region="US" has
> job="jenkins" and I watch to catch gaps here, it won't also fire in
> region="EU" which never has job="jenkins".... ?
>
> Thanks for any advice.
>
> Dan
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/91375704-0de4-4c1e-ada8-55ef78cc83b0n%40googlegroups.com.
