Hi,
As you had suggested, I tried to see it from the console. It was coming
as 0. So I triggered a request and then later on evaluating the same
expression, saw it coming as blank. But the other similar alerts were still
showing 0 on evaluating the expression and the alert was on. But as I said
I can see these metrics still coming even though the alert expressions
results in 0.
"alert_name": "test_alert",
"annotation_labelname": "Summary",
"annotation_labelvalue": "Test alert triggered",
"expr": "sum(increase(request_Count{key="clientId"}[24h])) < 1
and ON() hour() > 1 < 3",
"for": "30s"
And could you please share the same query with "up" ?
On Friday, August 13, 2021 at 12:20:40 PM UTC+5:30 Brian Candler wrote:
> > But defining the same expression in an alert, I am getting alerts
>
> Do you mean *not* getting alerts?
>
> You mentioned grafana; this makes me wonder are you using a grafana alert,
> instead of a prometheus alerting rule? If so, that's a grafana issue, not
> a prometheus one. But for now I'm going to assume you're talking about
> prometheus alerting rules. I also suggest you use prometheus' built-in
> query browser (typically at x.x.x.x:9090), rather than grafana, for testing.
>
> Any expression which is in the promQL browser which shows any value at all
> (even zero) generates an alert; when the graph is empty, there's no alert.
> Therefore, if you put
>
> *sum(increase(request_Count{key="clientId"}[24h])) < 1 and ON() hour() > 1
> < 3*
>
> into the promQL browser and select graph mode, do you see any lines? If
> so, you will get alerts. If you don't, then first check the prometheus
> console 'alerts' tab to see if the alert is firing there (just to ensure
> it's nothing to do with alertmanager not routing the alert properly) or at
> least is visible as an inactive rule (to ensure that prometheus has read
> this rule in). Other possible problems are that your rule is not being
> evaluated at a short enough interval, or that you have a "for:" value which
> means it has to trigger multiple times, taking longer than an hour. Since
> you didn't show your full alerting rule, I'm only speculating here.
>
> To simplify this problem, change your query to something you *know* has a
> value, e.g.
>
> up == 1 and ON() hour() > 1 < 3
>
> When I do this in the PromQL browser, set to 'graph' mode and set the
> duration to 1d or longer, then I can see the expression generating a value
> between 2am and 3am. Therefore if put into an alerting rule, it should
> also generate an alert overnight.
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/4c6624a9-2c2b-42d3-97f4-9e8ef3936964n%40googlegroups.com.
