Hi Everyone,
I deployed Prometheus, AlertManager as single instance on Kubernetes. I
facing issue - all alerts are resolved in the group if anyone alert in the
group is resolved but slack/webhook channels did receive notification for
one alert but not for all fired alerts in the group .
I am new to Prometheus/AlertManager and spent so much time for debugging
but no luck. So I came here to understand why resolved alerts are not
received by channels.
I have mentioned example below. Could you please help me what could be the
cause for this issue?
For example:- *alert1, alert2, alert3* are belonging to same group -
group_by [job, instance]. Here job is *engine_metrics_job* and instance is
*1000*.
*1:32 - alert2 - fired - received alerts *
* 3:42 - alert1, alert2 - fired - received alerts *
*3:47 - alert1 alert2, alert3 - fired - received alerts *
*4:02 - alert2 - resolved - received alerts*
The *alert1, alert3* alerts are cleared from AlertManager without sending
resolve alerts notification to slack/webhook channel.
AlertManager build version and configuration details below
*Alertmanager build version details:*
Status Uptime: 2022-05-23T23:35:53.161Z Cluster Status Status: disabled
Version Information Branch: HEAD BuildDate: 20191211-14:13:14 BuildUser:
root@00c3106655f8 GoVersion: go1.13.5 Revision:
f74be0400a6243d10bb53812d6fa408ad71ff32d Version: 0.20.0
*AlertManager configuration details:*
global: {} route: receiver: slack-alerts-1 # Fallback. group_wait: 30s
group_interval: 5m group_by: ['job', 'instance'] routes: - match: job:
engine_metrics_job receiver: slack-alerts-1 continue: true - match:
severity: page receiver: slack-alerts-2 continue: true - match_re: job:
engine_metrics_job receiver: webhook receivers: - name: slack-alerts-1
slack_configs: - send_resolved: true username: 'Prometheus Alertmanager'
channel: '#alerts-1' icon_url: '<icon_url>' api_url: '<slack_webhook-url>'
text: |- {{ range .Alerts }} *Alert:* {{ .Annotations.summary }} - `{{
.Labels.severity }}` *Description:* {{ .Annotations.description }}
*Details:* {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }} {{ end }} - name: slack-alerts-2 slack_configs: - send_resolved:
true username: 'Prometheus Alertmanager' channel: '#alerts-2' icon_url:
'<icon_url>' api_url: '<slack_webhook-url>' text: |- {{ range .Alerts }}
*Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }} *Details:* {{ range
.Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}` {{ end }} {{ end }}
- name: webhook webhook_configs: - send_resolved: true url:
'https://localhost:8180/service/event/webhook'
Thank you,
Shiva
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/87363086-8f9c-418f-9f53-d406b4a4076an%40googlegroups.com.
