See https://github.com/prometheus/alertmanager/issues/1944
A group_interval parameter of 1sec isn't really recommended and explains why Alertmanager sends emails so frequently. On Tue, Feb 18, 2020 at 1:25 PM Theofanis Pispirigkos <[email protected]> wrote: > > Hi , in my case a rendering error caused alertmanager to send multiple empty > emails. I am using prometheus operator with 2 alertmanagers in version v0.18.1 > The alertmanager configuration looks like: > > config: > route: > routes: > > - match: > severity: ticket > group_by: ['...'] > group_wait: 1s > group_interval: 1s > repeat_interval: 10d > receiver: ticket > > receivers: > > - name: 'ticket' > email_configs: > - to: '....' > headers: > subject: 'Prometheus alerts' > html: >- > {{ range .Alerts }} > <pre> > title: {{ .Annotations.summary }} ({{ .Labels.cluster }}) > assignee: {{ .Annotations.assignee }} > {{ end }} > send_resolved: true > > > The error occured when i used .GroupLabels.cluster instead of .Labels.cluster > . Alertmanager logged the bellow messages and sent multiple empty mails. > > level=error ts=2020-02-18T11:17:42.017Z caller=notify.go:367 > component=dispatcher msg="Error on notify" err="cancelling notify retry > for \"email\" due to unrecoverable error: execute html template: template: > :13:32: executing \"\" at <.GroupLabels.k8scluster>: can > 't evaluate field GroupLabels in type template.Alert" context_err=null > > level=error ts=2020-02-18T11:17:42.017Z caller=dispatch.go:266 > component=dispatcher msg="Notify for alerts failed" num_alerts=1 err= > "cancelling notify retry for \"email\" due to unrecoverable error: execute > html template: template: :13:32: executing \"\" at <.Grou > pLabels.k8scluster>: can't evaluate field GroupLabels in type template.Alert" > > level=error ts=2020-02-18T11:17:43.020Z caller=notify.go:367 > component=dispatcher msg="Error on notify" err="cancelling notify retry > for \"email\" due to unrecoverable error: execute html template: template: > :13:32: executing \"\" at <.GroupLabels.k8scluster>: can > 't evaluate field GroupLabels in type template.Alert" context_err=null > > level=error ts=2020-02-18T11:17:43.020Z caller=dispatch.go:266 > component=dispatcher msg="Notify for alerts failed" num_alerts=1 err= > "cancelling notify retry for \"email\" due to unrecoverable error: execute > html template: template: :13:32: executing \"\" at <.Grou > pLabels.k8scluster>: can't evaluate field GroupLabels in type template.Alert" > > level=error ts=2020-02-18T11:17:44.020Z caller=notify.go:367 > component=dispatcher msg="Error on notify" err="cancelling notify retry > for \"email\" due to unrecoverable error: execute html template: template: > :13:32: executing \"\" at <.GroupLabels.k8scluster>: can > 't evaluate field GroupLabels in type template.Alert" context_err=null > > level=error ts=2020-02-18T11:17:44.020Z caller=dispatch.go:266 > component=dispatcher msg="Notify for alerts failed" num_alerts=1 err= > "cancelling notify retry for \"email\" due to unrecoverable error: execute > html template: template: :13:32: executing \"\" at <.Grou > pLabels.k8scluster>: can't evaluate field GroupLabels in type template.Alert" > > Since the emails are routed to a ticketing system , having the alertmanager > spamming with empty emails is not an acceptable behavior. So my questions is > why alertmanager sent eventually the email and why it kept firing emails > until the time i fixed the configuration. Is there any way to prevent this > situation from happening ? > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/eca83f70-7ab0-42c3-8ce1-887285146323%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAM6RFu4rXc--WHQ71S_N3am9XCbBgvVg6W80JRVR8KqKt1bbGA%40mail.gmail.com.
