> even if application is not down, it sends alerts every 1 min. how to
debug this i am using below exp:- alert: "Instance Down" expr: up == 0
You need to show the actual alerts, from the Prometheus web interface
and/or the notifications, and then describe how these are different from
what you expect.
I very much doubt that the expression "up == 0" is firing unless there is
at least one target which is not being scraped, and therefore the "up"
metric has a value of 0 for a particular timeseries (metric with a given
set of labels).
> if the threshold cross and value changes, it fires multiple alerts having
same alert rule thats fine. But with same '{{ $value }}' it should fire
alerts after 5 min. same alert rule with same value should not get fire for
next 5 min. how to get this ??
I cannot work out what problem you are trying to describe. As long as you
only use '{{ $value }}' in annotations, not labels, then the same alert
will just continue firing.
Whether you get repeated *notifications* about that ongoing alert is a
different matter. With "repeat_interval: 15m" you should get them every 15
minutes at least. You may get additional notifications if a new alert is
added into the same alert group, or one is resolved from the alert group.
> whats is for, keep_firing_for and evaluation_interval ?
keep_firing_for is debouncing: once the alert condition has gone away, it
will continue firing for this period of time. This is so that if the alert
condition vanishes briefly but reappears, it doesn't cause the alert to be
resolved and then retriggered.
evaluation_interval is how often the alerting expression is evaluated.
On Friday, 14 February 2025 at 15:53:24 UTC Amol Nagotkar wrote:
> Hi all,
> i want same alert(alert rule) to be fire after 5 min, currently i am
> getting same alert (alert rule) after every one minute for same '{{ $value
> }}'.
> if the threshold cross and value changes, it fires multiple alerts having
> same alert rule thats fine. But with same '{{ $value }}' it should fire
> alerts after 5 min. same alert rule with same value should not get fire for
> next 5 min. how to get this ??
> even if application is not down, it sends alerts every 1 min. how to debug
> this i am using below exp:- alert: "Instance Down" expr: up == 0
> whats is for, keep_firing_for and evaluation_interval ?
> prometheus.yml
>
> global:
> scrape_interval: 15s # Set the scrape interval to every 15 seconds.
> Default is every 1 minute.
> evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is
> every 1 minute.
>
> alerting:
> alertmanagers:
>
> - static_configs:
> - targets:
> - ip:port
>
> rule_files:
>
> - "alerts_rules.yml"
>
> scrape_configs:
>
> - job_name: "prometheus"
> static_configs:
> - targets: ["ip:port"]
>
> alertmanager.yml
> global:
> resolve_timeout: 5m
> route:
> group_wait: 5s
> group_interval: 5m
> repeat_interval: 15m
> receiver: webhook_receiver
> receivers:
>
> - name: webhook_receiver
> webhook_configs:
> - url: 'http://ip:port'
> send_resolved: false
>
> alerts_rules.yml
>
>
> groups:
> - name: instance_alerts
> rules:
> - alert: "Instance Down"
> expr: up == 0
> # for: 30s
> # keep_firing_for: 30s
> labels:
> severity: "Critical"
> annotations:
> summary: "Endpoint {{ $labels.instance }} down"
> description: "{{ $labels.instance }} of job {{ $labels.job }} has
> been down for more than 30 sec."
>
> - name: rabbitmq_alerts
> rules:
> - alert: "Consumer down for last 1 min"
> expr: rabbitmq_queue_consumers == 0
> # for: 1m
> # keep_firing_for: 30s
> labels:
> severity: Critical
> annotations:
> summary: "shortify | '{{ $labels.queue }}' has no consumers"
> description: "The queue '{{ $labels.queue }}' in vhost '{{
> $labels.vhost }}' has zero consumers for more than 30 sec. Immediate
> attention is required."
>
>
> - alert: "Total Messages > 10k in last 1 min"
> expr: rabbitmq_queue_messages > 10000
> # for: 1m
> # keep_firing_for: 30s
> labels:
> severity: Critical
> annotations:
> summary: "'{{ $labels.queue }}' has total '{{ $value }}' messages
> for more than 1 min."
> description: |
> Queue {{ $labels.queue }} in RabbitMQ has total {{ $value }}
> messages for more than 1 min.
>
>
> Thank you in advance.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/ff023f47-09d3-4bbd-864f-5bb2f8fdc4e6n%40googlegroups.com.
