Rather than use absent(), you can use the Prometheus metamon metric prometheus_target_scrape_pool_targets.
Prometheus alerts are meant to be done in layers, where you have separate alerts on `jenkins_up`, `up`, and `prometheus_target_scrape_pool_targets`. Trying to manipulate alerts with absent() tends to behave badly. On Sun, Oct 3, 2021 at 9:21 PM Brian Candler <[email protected]> wrote: > Or slightly weirder: > > absent(jenkins_up) and absent(absent(up{job="jenkins"})) > > absent(absent(...)) being a way to get the RHS to have no labels, to match > the LHS. > > On Sunday, 3 October 2021 at 20:14:59 UTC+1 Brian Candler wrote: > >> This might be an XY problem, because it is often better to have a defined >> "up/down" metric (with value 1/0), which tells you whether something worked >> or not, rather than alerting on presence or absence of a metric. >> >> However, to answer your question directly, I think you would need to >> include some condition saying whether that metric *should* be there or not >> - which is the presence of some other metric. The "up" metric added by all >> scrape jobs can be useful for this. In this case, I expect >> *up{job="jenkins"}* will exist, if and only if you have a 'jenkins' >> scrape job in that region. Therefore maybe something like this will do >> what you want: >> >> absent(jenkins_up{job="jenkins"}) unless on (job) >> absent(up{job="jenkins"}) >> >> which I think may simplify, if the 'jenkins_up' metric is only scraped by >> the 'jenkins' job, to this (not sure): >> >> absent(jenkins_up) unless on () absent(up{job="jenkins"}) >> >> On Sunday, 3 October 2021 at 19:40:23 UTC+1 Dan S wrote: >> >>> Hi, >>> >>> Looking for some general advice about shared prom alert rules between >>> regions. We currently push the same alert rules to all regions, and >>> sometimes we run into situations where we have a specific job in region X >>> but not Y. >>> >>> This is fine for basic cases, such as *up{job="jenkins"} == 0* which >>> will be ignored in regions where there's no jenkins job present (or could >>> easily specify region="X"). >>> >>> But in some situations I'd like to use absent on a metric that often has >>> gaps for example >>> *absent(jenkins_up{job="jenkins"})* >>> This would trigger in all regions, whether or not there's a job >>> "jenkins" (obviously because it's triggering on the missing metrics) even >>> if I try to be more specific: *absent(jenkins_up{job="jenkins", >>> region="US"}).* >>> >>> Any suggestions how I can craft an alert query using absent() in on >>> metrics that don't appear in all regions? So that if region="US" has >>> job="jenkins" and I watch to catch gaps here, it won't also fire in >>> region="EU" which never has job="jenkins".... ? >>> >>> Thanks for any advice. >>> >>> Dan >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/8533225b-267c-4dd1-af32-5967966d5156n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/8533225b-267c-4dd1-af32-5967966d5156n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqNZ4REUnhaMxoC%2B%2BcnFBdUHOPnOqKC_f7TfLTOkHJw0Q%40mail.gmail.com.
