To add, usually this request happens around 16:00 UTC. Not sure if that
causes the issue.
On Friday, August 13, 2021 at 1:22:31 PM UTC+5:30 Govind Madhu wrote:
> Hi,
>
> As you had suggested, I tried to see it from the console. It was coming
> as 0. So I triggered a request and then later on evaluating the same
> expression, saw it coming as blank. But the other similar alerts were still
> showing 0 on evaluating the expression and the alert was on. But as I said
> I can see these metrics still coming even though the alert expressions
> results in 0.
>
> "alert_name": "test_alert",
> "annotation_labelname": "Summary",
> "annotation_labelvalue": "Test alert triggered",
> "expr": "sum(increase(request_Count{key="clientId"}[24h])) < 1
> and ON() hour() > 1 < 3",
> "for": "30s"
>
> And could you please share the same query with "up" ?
> On Friday, August 13, 2021 at 12:20:40 PM UTC+5:30 Brian Candler wrote:
>
>> > But defining the same expression in an alert, I am getting alerts
>>
>> Do you mean *not* getting alerts?
>>
>> You mentioned grafana; this makes me wonder are you using a grafana
>> alert, instead of a prometheus alerting rule? If so, that's a grafana
>> issue, not a prometheus one. But for now I'm going to assume you're
>> talking about prometheus alerting rules. I also suggest you use
>> prometheus' built-in query browser (typically at x.x.x.x:9090), rather than
>> grafana, for testing.
>>
>> Any expression which is in the promQL browser which shows any value at
>> all (even zero) generates an alert; when the graph is empty, there's no
>> alert. Therefore, if you put
>>
>> *sum(increase(request_Count{key="clientId"}[24h])) < 1 and ON() hour() >
>> 1 < 3*
>>
>> into the promQL browser and select graph mode, do you see any lines? If
>> so, you will get alerts. If you don't, then first check the prometheus
>> console 'alerts' tab to see if the alert is firing there (just to ensure
>> it's nothing to do with alertmanager not routing the alert properly) or at
>> least is visible as an inactive rule (to ensure that prometheus has read
>> this rule in). Other possible problems are that your rule is not being
>> evaluated at a short enough interval, or that you have a "for:" value which
>> means it has to trigger multiple times, taking longer than an hour. Since
>> you didn't show your full alerting rule, I'm only speculating here.
>>
>> To simplify this problem, change your query to something you *know* has a
>> value, e.g.
>>
>> up == 1 and ON() hour() > 1 < 3
>>
>> When I do this in the PromQL browser, set to 'graph' mode and set the
>> duration to 1d or longer, then I can see the expression generating a value
>> between 2am and 3am. Therefore if put into an alerting rule, it should
>> also generate an alert overnight.
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/4f8e88b7-7b89-4541-ac05-99fdd018ca50n%40googlegroups.com.
