I added AlertManager Build version and configuration details:
*AlertManager Build version:*
Status Uptime:
2022-05-23T23:35:53.161Z
Cluster Status
Status: disabled
Version Information
Branch: HEAD
BuildDate: 20191211-14:13:14
BuildUser: root@00c3106655f8
GoVersion: go1.13.5
Revision: f74be0400a6243d10bb53812d6fa408ad71ff32d
Version: 0.20.0
```
*AlertManager configuration details:*
global: {}
route:
receiver: slack-alerts-1 # Fallback.
group_wait: 30s
group_interval: 5m
group_by: ['job', 'instance']
routes:
- match:
job: engine_metrics_job
receiver: slack-alerts-1
continue: true
- match:
severity: page
receiver: slack-alerts-2
continue: true
- match_re:
job: engine_metrics_job
receiver: webhook
receivers:
- name: slack-alerts-1
slack_configs:
- send_resolved: true
username: 'Prometheus Alertmanager'
channel: '#alerts-1'
icon_url: '<icon_url>'
api_url: '<slack_webhook-url>'
text: |-
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }}
*Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}
{{ end }}
- name: slack-alerts-2
slack_configs:
- send_resolved: true
username: 'Prometheus Alertmanager'
channel: '#alerts-2'
icon_url: '<icon_url>'
api_url: '<slack_webhook-url>'
text: |-
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }}
*Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}
{{ end }}
- name: webhook
webhook_configs:
- send_resolved: true
url: 'https://localhost:8180/service/event/webhook'
Thank you,
Shiva
On Monday, June 13, 2022 at 9:30:53 AM UTC+5:30 shivakumar sajjan wrote:
> Hi Everyone,
>
> I deployed Prometheus, AlertManager as single instance on Kubernetes. I
> facing issue - all alerts are resolved in the group if anyone alert in the
> group is resolved but slack/webhook channels did receive notification for
> one alert but not for all fired alerts in the group .
>
> I am new to Prometheus/AlertManager and spent so much time for debugging
> but no luck. So I came here to understand why resolved alerts are not
> received by channels.
>
> I have mentioned example below. Could you please help me what could be the
> cause for this issue?
>
> For example:- *alert1, alert2, alert3* are belonging to same group -
> group_by [job, instance]. Here job is *engine_metrics_job* and instance
> is *1000*.
>
> *1:32 - alert2 - fired - received alerts *
>
> * 3:42 - alert1, alert2 - fired - received alerts *
>
> *3:47 - alert1 alert2, alert3 - fired - received alerts *
>
> *4:02 - alert2 - resolved - received alerts*
>
>
> The *alert1, alert3* alerts are cleared from AlertManager without sending
> resolve alerts notification to slack/webhook channel.
>
> AlertManager build version and configuration details below
>
> *Alertmanager build version details:*
> Status Uptime: 2022-05-23T23:35:53.161Z Cluster Status Status: disabled
> Version Information Branch: HEAD BuildDate: 20191211-14:13:14 BuildUser:
> root@00c3106655f8 GoVersion: go1.13.5 Revision:
> f74be0400a6243d10bb53812d6fa408ad71ff32d Version: 0.20.0
>
> *AlertManager configuration details:*
> global: {} route: receiver: slack-alerts-1 # Fallback. group_wait: 30s
> group_interval: 5m group_by: ['job', 'instance'] routes: - match: job:
> engine_metrics_job receiver: slack-alerts-1 continue: true - match:
> severity: page receiver: slack-alerts-2 continue: true - match_re: job:
> engine_metrics_job receiver: webhook receivers: - name: slack-alerts-1
> slack_configs: - send_resolved: true username: 'Prometheus Alertmanager'
> channel: '#alerts-1' icon_url: '<icon_url>' api_url: '<slack_webhook-url>'
> text: |- {{ range .Alerts }} *Alert:* {{ .Annotations.summary }} - `{{
> .Labels.severity }}` *Description:* {{ .Annotations.description }}
> *Details:* {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
> {{ end }} {{ end }} - name: slack-alerts-2 slack_configs: - send_resolved:
> true username: 'Prometheus Alertmanager' channel: '#alerts-2' icon_url:
> '<icon_url>' api_url: '<slack_webhook-url>' text: |- {{ range .Alerts }}
> *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
> *Description:* {{ .Annotations.description }} *Details:* {{ range
> .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}` {{ end }} {{ end }}
> - name: webhook webhook_configs: - send_resolved: true url: '
> https://localhost:8180/service/event/webhook'
>
>
> Thank you,
>
> Shiva
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/e2ab20dd-94ac-4453-8cdc-f65819d1e573n%40googlegroups.com.
