That will never work. All labels (including severity) are things set in your alerting rules. You will never see a label with severity=resolved, unless you actually make an alerting rule which sets severity=resolved when it fires.
(match_re does work though; it's the old syntax whereas "matchers" is the new and preferred syntax) Moral: never trust what ChatGPT tells you. Off the top of my head, I can't think of any way to send a message when alerts are resolved but *not* when alerts are firing. There are good reasons for not sending *any* resolved messages though: https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped On Sunday, 18 June 2023 at 12:11:31 UTC Kolja Krückmann wrote: > Hi Brian, thanks here are the routes to the receivers: > > routes: > - receiver: email_critical > group_interval: 1m > group_wait: 5m > repeat_interval: 24h > group_by: ['alertname', 'severity'] > matchers: > - severity="critical" > - severity!="resolved" > > - receiver: email_warning > group_interval: 10m > group_wait: 5m > repeat_interval: 24h > group_by: ['alertname', 'severity'] > matchers: > - severity="warning" > - severity!="resolved" > > - receiver: email_resolved > match_re: > severity: ^(resolved)$ > > > the match_re part from resolved is from good ol' friend chatgpt > > Brian Candler schrieb am Samstag, 17. Juni 2023 um 00:48:14 UTC+2: > >> You'll need to show the rest of your alertmanager config. "Receivers" by >> themselves don't do anything; you have to refer to them in routing rules. >> >> On Friday, 16 June 2023 at 09:23:56 UTC Kolja Krückmann wrote: >> >>> Or do I need to change it to explicitly say in the crit and warn mail to >>> range over the .Alerts.Firing in order to get the third receiver to work? >>> >>> receivers: >>> - name: 'email' >>> email_configs: >>> - to: '[email protected]' >>> send_resolved: false >>> headers: >>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{ >>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}' >>> >>> >>> - name: 'email_warning' >>> email_configs: >>> - to: '[email protected]' >>> send_resolved: false >>> headers: >>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ >>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}' >>> >>> >>> - name: 'email_resolved' >>> email_configs: >>> - to: '[email protected]' >>> send_resolved: true >>> headers: >>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ >>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}' >>> >>> Kolja Krückmann schrieb am Freitag, 16. Juni 2023 um 11:21:59 UTC+2: >>> >>>> I changed the config to this: >>>> >>>> - name: 'email_resolved' >>>> email_configs: >>>> - to: '[email protected]' >>>> send_resolved: true >>>> headers: >>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ >>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}' >>>> >>>> But this is sending absolutly nothing. What did I do wrong? >>>> >>>> Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2: >>>> >>>>> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert >>>>> group is likely to contain a mixture of firing alerts and resolved alerts. >>>>> >>>>> Also, note that your "send_resolved: true" receiver doesn't only send >>>>> resolved messages: it will send on firing *and* on resolved. Hence the >>>>> subject "Resolved: ..." is likely to be very misleading. >>>>> >>>>> On Friday, 9 June 2023 at 13:42:13 UTC+1 Kolja Krückmann wrote: >>>>> >>>>>> Got it working like this: >>>>>> >>>>>> receivers: >>>>>> - name: 'email' >>>>>> email_configs: >>>>>> - to: '[email protected]' >>>>>> send_resolved: false >>>>>> headers: >>>>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: >>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>> >>>>>> - name: 'email_warning' >>>>>> email_configs: >>>>>> - to: '[email protected]' >>>>>> send_resolved: false >>>>>> headers: >>>>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ >>>>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>> >>>>>> - name: 'email_resolved' >>>>>> email_configs: >>>>>> - to: '[email protected]' >>>>>> send_resolved: true >>>>>> headers: >>>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: >>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>> >>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 UTC+2: >>>>>> >>>>>>> [EDIT] >>>>>>> Also I want to split the firing and resolved alerts. >>>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 >>>>>>> UTC+2: >>>>>>> >>>>>>>> Hi all, I tried it now for two days straight and I just don't get >>>>>>>> it working. >>>>>>>> >>>>>>>> First of all where do I actually find the expressions I can use to >>>>>>>> define the subject under the header? I am still missing a template I >>>>>>>> can >>>>>>>> use? I don't really have one in my Prom/Alert-files. >>>>>>>> Second of all my current receivers are configured like so: >>>>>>>> >>>>>>>> receivers: >>>>>>>> - name: 'email' >>>>>>>> email_configs: >>>>>>>> - to: '[email protected]' >>>>>>>> send_resolved: true >>>>>>>> headers: >>>>>>>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{ >>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>> >>>>>>>> Like Brian said, I want to iterate in the second half of that >>>>>>>> "statement" (I don't actually know how these {{ .xyz }} are called) >>>>>>>> through >>>>>>>> all instances and the first one should just say which alert is now >>>>>>>> firing. >>>>>>>> I also tried it as followed: >>>>>>>> >>>>>>>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{ >>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>> and >>>>>>>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{ >>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>> >>>>>>>> but nothing is really working. >>>>>>>> >>>>>>>> All I really want is the Subject like so e.x.: >>>>>>>> InstanceDown - SVR-DS01 SVR-DC11 >>>>>>>> ^^^^^^^^ ^^^^^^^^ ^^^^^^^^ >>>>>>>> Alertname - Instance1 Instance2 (if it is also possible to >>>>>>>> seperate the instances via comma would be perfect) >>>>>>>> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2: >>>>>>>> >>>>>>>>> As its name implies, "CommonLabels" contains only those labels >>>>>>>>> which are common to all alerts in the group. If there are multiple >>>>>>>>> instances, then the "instance" label is not common to all alerts, so >>>>>>>>> won't >>>>>>>>> be in this object. >>>>>>>>> >>>>>>>>> You want to iterate over "Alerts" (or "Alerts.Firing") and look at >>>>>>>>> the Labels.instance within each alert. You will find an example >>>>>>>>> showing how >>>>>>>>> to do that here: >>>>>>>>> >>>>>>>>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts >>>>>>>>> >>>>>>>>> Additional references: >>>>>>>>> https://prometheus.io/docs/alerting/latest/notifications/#data >>>>>>>>> >>>>>>>>> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/ >>>>>>>>> >>>>>>>>> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote: >>>>>>>>> >>>>>>>>>> Currently my alertmanager.yml looks like this: >>>>>>>>>> >>>>>>>>>> receivers: >>>>>>>>>> - name: 'email' >>>>>>>>>> email_configs: >>>>>>>>>> - to: '[email protected]' >>>>>>>>>> send_resolved: true >>>>>>>>>> headers: >>>>>>>>>> Subject: '{{ .CommonLabels.alertname }} - {{ >>>>>>>>>> .CommonLabels.instance }}' >>>>>>>>>> >>>>>>>>>> Unfortunately the subject in the mail now only displays both >>>>>>>>>> labels if only there is an alert for a single instance (like High >>>>>>>>>> CPU Usage >>>>>>>>>> on core 0,1,2,3 on one instance (see screenshot attatched)) As soon >>>>>>>>>> as the >>>>>>>>>> alert is for more then just one instance the instance lable is >>>>>>>>>> completly >>>>>>>>>> missing... Why is it missing? And how can I change it to display all >>>>>>>>>> instances affected by that alert. >>>>>>>>>> >>>>>>>>>> Kind regards. >>>>>>>>>> Kolja >>>>>>>>>> >>>>>>>>>> [image: mail.png] >>>>>>>>>> >>>>>>>>>> Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 UTC+2: >>>>>>>>>> >>>>>>>>>> Hi y'all >>>>>>>>>> I'm trying to change the way the mail from my alertmanger looks >>>>>>>>>> and behaves. >>>>>>>>>> My goal is to have the subject list the alertname and the targets >>>>>>>>>> detected by that alert. (I need it like that for future todo's) >>>>>>>>>> My first problem is, i dont have any templates in my files. Do I >>>>>>>>>> need to clone one from git? How does the prom know, I have a >>>>>>>>>> template? >>>>>>>>>> The second one would be to change the subject in order for my >>>>>>>>>> requirements to fit. >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> Kolja >>>>>>>>>> >>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e77a8e5e-d448-4c04-aa5b-d4f160ed9ed5n%40googlegroups.com.
