> But defining the same expression in an alert, I am getting alerts
Do you mean *not* getting alerts?
You mentioned grafana; this makes me wonder are you using a grafana alert,
instead of a prometheus alerting rule? If so, that's a grafana issue, not
a prometheus one. But for now I'm going to assume you're talking about
prometheus alerting rules. I also suggest you use prometheus' built-in
query browser (typically at x.x.x.x:9090), rather than grafana, for testing.
Any expression which is in the promQL browser which shows any value at all
(even zero) generates an alert; when the graph is empty, there's no alert.
Therefore, if you put
*sum(increase(request_Count{key="clientId"}[24h])) < 1 and ON() hour() > 1
< 3*
into the promQL browser and select graph mode, do you see any lines? If
so, you will get alerts. If you don't, then first check the prometheus
console 'alerts' tab to see if the alert is firing there (just to ensure
it's nothing to do with alertmanager not routing the alert properly) or at
least is visible as an inactive rule (to ensure that prometheus has read
this rule in). Other possible problems are that your rule is not being
evaluated at a short enough interval, or that you have a "for:" value which
means it has to trigger multiple times, taking longer than an hour. Since
you didn't show your full alerting rule, I'm only speculating here.
To simplify this problem, change your query to something you *know* has a
value, e.g.
up == 1 and ON() hour() > 1 < 3
When I do this in the PromQL browser, set to 'graph' mode and set the
duration to 1d or longer, then I can see the expression generating a value
between 2am and 3am. Therefore if put into an alerting rule, it should
also generate an alert overnight.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/6e0a2c37-dbfa-4f4b-8d33-13c23d74a81an%40googlegroups.com.
