A query expression which generates *any* value - even zero - will trigger
an alert. It is the presence of a value, not the actual value, which fires
it.
If you consider the expression
foo > 4
this is a filter. If the value of foo is 3 then it has no value, and if
the value of foo is 5, then it has value 5. It doesn't return a boolean
true or false.
There is another expression: foo > bool 4, which returns 0 or 1. But that
would trigger an alert continuously, as long as there is any metric "foo".
See:
https://prometheus.io/docs/prometheus/latest/querying/operators/#comparison-binary-operators
On Friday, 13 August 2021 at 08:52:31 UTC+1 Govind Madhu wrote:
> Hi,
>
> As you had suggested, I tried to see it from the console. It was coming
> as 0. So I triggered a request and then later on evaluating the same
> expression, saw it coming as blank. But the other similar alerts were still
> showing 0 on evaluating the expression and the alert was on. But as I said
> I can see these metrics still coming even though the alert expressions
> results in 0.
>
> "alert_name": "test_alert",
> "annotation_labelname": "Summary",
> "annotation_labelvalue": "Test alert triggered",
> "expr": "sum(increase(request_Count{key="clientId"}[24h])) < 1
> and ON() hour() > 1 < 3",
> "for": "30s"
>
> And could you please share the same query with "up" ?
> On Friday, August 13, 2021 at 12:20:40 PM UTC+5:30 Brian Candler wrote:
>
>> > But defining the same expression in an alert, I am getting alerts
>>
>> Do you mean *not* getting alerts?
>>
>> You mentioned grafana; this makes me wonder are you using a grafana
>> alert, instead of a prometheus alerting rule? If so, that's a grafana
>> issue, not a prometheus one. But for now I'm going to assume you're
>> talking about prometheus alerting rules. I also suggest you use
>> prometheus' built-in query browser (typically at x.x.x.x:9090), rather than
>> grafana, for testing.
>>
>> Any expression which is in the promQL browser which shows any value at
>> all (even zero) generates an alert; when the graph is empty, there's no
>> alert. Therefore, if you put
>>
>> *sum(increase(request_Count{key="clientId"}[24h])) < 1 and ON() hour() >
>> 1 < 3*
>>
>> into the promQL browser and select graph mode, do you see any lines? If
>> so, you will get alerts. If you don't, then first check the prometheus
>> console 'alerts' tab to see if the alert is firing there (just to ensure
>> it's nothing to do with alertmanager not routing the alert properly) or at
>> least is visible as an inactive rule (to ensure that prometheus has read
>> this rule in). Other possible problems are that your rule is not being
>> evaluated at a short enough interval, or that you have a "for:" value which
>> means it has to trigger multiple times, taking longer than an hour. Since
>> you didn't show your full alerting rule, I'm only speculating here.
>>
>> To simplify this problem, change your query to something you *know* has a
>> value, e.g.
>>
>> up == 1 and ON() hour() > 1 < 3
>>
>> When I do this in the PromQL browser, set to 'graph' mode and set the
>> duration to 1d or longer, then I can see the expression generating a value
>> between 2am and 3am. Therefore if put into an alerting rule, it should
>> also generate an alert overnight.
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/45989eb4-cf27-4078-baed-22a999c635c1n%40googlegroups.com.
