Hi all,
i want same alert(alert rule) to be fire after 5 min, currently i am
getting same alert (alert rule) after every one minute for same '{{ $value
}}'.
if the threshold cross and value changes, it fires multiple alerts having
same alert rule thats fine. But with same '{{ $value }}' it should fire
alerts after 5 min. same alert rule with same value should not get fire for
next 5 min. how to get this ??
even if application is not down, it sends alerts every 1 min. how to debug
this i am using below exp:- alert: "Instance Down" expr: up == 0
whats is for, keep_firing_for and evaluation_interval ?
prometheus.yml
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default
is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is
every 1 minute.
alerting:
alertmanagers:
- static_configs:
- targets:
- ip:port
rule_files:
- "alerts_rules.yml"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["ip:port"]
alertmanager.yml
global:
resolve_timeout: 5m
route:
group_wait: 5s
group_interval: 5m
repeat_interval: 15m
receiver: webhook_receiver
receivers:
- name: webhook_receiver
webhook_configs:
- url: 'http://ip:port'
send_resolved: false
alerts_rules.yml
groups:
- name: instance_alerts
rules:
- alert: "Instance Down"
expr: up == 0
# for: 30s
# keep_firing_for: 30s
labels:
severity: "Critical"
annotations:
summary: "Endpoint {{ $labels.instance }} down"
description: "{{ $labels.instance }} of job {{ $labels.job }} has
been down for more than 30 sec."
- name: rabbitmq_alerts
rules:
- alert: "Consumer down for last 1 min"
expr: rabbitmq_queue_consumers == 0
# for: 1m
# keep_firing_for: 30s
labels:
severity: Critical
annotations:
summary: "shortify | '{{ $labels.queue }}' has no consumers"
description: "The queue '{{ $labels.queue }}' in vhost '{{
$labels.vhost }}' has zero consumers for more than 30 sec. Immediate
attention is required."
- alert: "Total Messages > 10k in last 1 min"
expr: rabbitmq_queue_messages > 10000
# for: 1m
# keep_firing_for: 30s
labels:
severity: Critical
annotations:
summary: "'{{ $labels.queue }}' has total '{{ $value }}' messages
for more than 1 min."
description: |
Queue {{ $labels.queue }} in RabbitMQ has total {{ $value }}
messages for more than 1 min.
Thank you in advance.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/6a32aaa6-ae70-4eac-b3ff-f104f84180aen%40googlegroups.com.
