And additionally, when I evaluate the expression
request_Count{key="clientId"}[24h], I can see the value coming as 2
On Friday, August 13, 2021 at 1:24:32 PM UTC+5:30 Govind Madhu wrote:
> To add, usually this request happens around 16:00 UTC. Not sure if that
> causes the issue.
>
> On Friday, August 13, 2021 at 1:22:31 PM UTC+5:30 Govind Madhu wrote:
>
>> Hi,
>>
>> As you had suggested, I tried to see it from the console. It was coming
>> as 0. So I triggered a request and then later on evaluating the same
>> expression, saw it coming as blank. But the other similar alerts were still
>> showing 0 on evaluating the expression and the alert was on. But as I said
>> I can see these metrics still coming even though the alert expressions
>> results in 0.
>>
>> "alert_name": "test_alert",
>> "annotation_labelname": "Summary",
>> "annotation_labelvalue": "Test alert triggered",
>> "expr": "sum(increase(request_Count{key="clientId"}[24h])) < 1
>> and ON() hour() > 1 < 3",
>> "for": "30s"
>>
>> And could you please share the same query with "up" ?
>> On Friday, August 13, 2021 at 12:20:40 PM UTC+5:30 Brian Candler wrote:
>>
>>> > But defining the same expression in an alert, I am getting alerts
>>>
>>> Do you mean *not* getting alerts?
>>>
>>> You mentioned grafana; this makes me wonder are you using a grafana
>>> alert, instead of a prometheus alerting rule? If so, that's a grafana
>>> issue, not a prometheus one. But for now I'm going to assume you're
>>> talking about prometheus alerting rules. I also suggest you use
>>> prometheus' built-in query browser (typically at x.x.x.x:9090), rather than
>>> grafana, for testing.
>>>
>>> Any expression which is in the promQL browser which shows any value at
>>> all (even zero) generates an alert; when the graph is empty, there's no
>>> alert. Therefore, if you put
>>>
>>> *sum(increase(request_Count{key="clientId"}[24h])) < 1 and ON() hour() >
>>> 1 < 3*
>>>
>>> into the promQL browser and select graph mode, do you see any lines? If
>>> so, you will get alerts. If you don't, then first check the prometheus
>>> console 'alerts' tab to see if the alert is firing there (just to ensure
>>> it's nothing to do with alertmanager not routing the alert properly) or at
>>> least is visible as an inactive rule (to ensure that prometheus has read
>>> this rule in). Other possible problems are that your rule is not being
>>> evaluated at a short enough interval, or that you have a "for:" value which
>>> means it has to trigger multiple times, taking longer than an hour. Since
>>> you didn't show your full alerting rule, I'm only speculating here.
>>>
>>> To simplify this problem, change your query to something you *know* has
>>> a value, e.g.
>>>
>>> up == 1 and ON() hour() > 1 < 3
>>>
>>> When I do this in the PromQL browser, set to 'graph' mode and set the
>>> duration to 1d or longer, then I can see the expression generating a value
>>> between 2am and 3am. Therefore if put into an alerting rule, it should
>>> also generate an alert overnight.
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7a65f3eb-bb64-4154-95de-db9bd7584ea3n%40googlegroups.com.
