I think it helps to think about Alertmanager webhooks differently. Alertmanager does not notify about individual alerts but about *groups* of alerts. These groups come into being, the number of alert instances in them potentially changes over time. Subsequent webhooks about the same groups are updates about the status, not a separate instance of anything. The resolution notification closes the group.
By design, you cannot rely on 1 webhook call = 1 alert. To ensure delivery, Alertmanager will err on the side of notifying more than needed if it is not sure what you already got. This is especially the case with clustered Alertmanager. The webhooks contain a groupKey field. Use this field to identify which group a notification is about, and update that one if you already have it in your UI. That way, there is only one thing to close as well. /MR On Tue, 3 Mar 2020, 16:54 EnthuDeveloper, <[email protected]> wrote: > Hi , > I am just curious to know if we have a custom webhook implemented to > receive alerts from Prometheus Alertmanager then what should be the > considerations for implementing the custom logic for resolved alerts ? > My question is more from a standpoint that if repeat_interval is > configured to be a frequent interval , then our system might have received > multiple alerts with firing status for the same alert. So when Prometheus > identifies that alert should be resolved now would it just send one alert > with resolved status ? > > My confusion is shouldn’t we mark all the existing alert instances with > matching alert name as resolved in our custom logic once Prometheus sees an > alert transitioning from firing to resolved status. > > Note : This logic is needed on our custom framework side to avoid listing > resolved alerts on the alert dashboard. > > > Any input would be greatly appreciated. > > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/ceaf9c38-3b17-47e0-a4bd-e79e7d229e3f%40googlegroups.com > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAFU3N5W5sOy-fy4ZyrGq%2BZw7b9-aNGjKe2CNy%3DjnqMr4eFQsLQ%40mail.gmail.com.
