Here are a couple of things to help understand why the original expression doesn't work as expected.
1. The comparison operators are filters, returning a vector of 0 or more elements which is subset of all the available timeseries, not a boolean value. (foo > my_custom_threshold) is comparing an instant vector with another instant vector. It only gives results where foo and my_custom_threshold have exactly matching label sets *and* the value of foo is greater than the corresponding value of my_custom_threshold. The values in the result vector are the values of the LHS. (foo > 0.7) is comparing an instant vector with a scalar. It gives results for *every* metric foo whose value is > 0.7. 2. (foo > x) OR (foo > y) isn't a boolean expression, it's a set (union) operation. (foo > x) gives all those timeseries whose metric name is foo and value is > x (foo > y) gives all those timeseries whose metric name is foo and value is > y (foo > x) OR (foo > y) will give you all the metrics from the LHS, *plus* all metrics on the RHS which don't have any matching label set on the LHS. The behaviour of this effectively is "foo > min(x,y)" [although that is not valid PromQL]. This is why you'll always get an alert for value over 0.7; if you set my_custom_threshold higher, meaning that the LHS doesn't give a result, the RHS will fill in the gap. I find that using the PromQL query interface in the prometheus web interface is very useful for graphing the expressions or subexpressions. Once you realise that "foo > 0.7" is basically just showing you the graph of foo, but with gaps where its value falls below 0.7, suddenly things become a lot clearer. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/90318052-a5c7-432d-a365-303842ad016e%40googlegroups.com.
