Thank you very much Brian! This will be my last post for now. I'm going to change company. I hope you can help my collegues as good as you helped me! This community is great, keep it up. Brian Candler schrieb am Sonntag, 18. Juni 2023 um 19:04:36 UTC+2:
> That will never work. All labels (including severity) are things set in > your alerting rules. You will never see a label with severity=resolved, > unless you actually make an alerting rule which sets severity=resolved when > it fires. > > (match_re does work though; it's the old syntax whereas "matchers" is the > new and preferred syntax) > > Moral: never trust what ChatGPT tells you. > > Off the top of my head, I can't think of any way to send a message when > alerts are resolved but *not* when alerts are firing. > > There are good reasons for not sending *any* resolved messages though: > > https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped > > On Sunday, 18 June 2023 at 12:11:31 UTC Kolja Krückmann wrote: > >> Hi Brian, thanks here are the routes to the receivers: >> >> routes: >> - receiver: email_critical >> group_interval: 1m >> group_wait: 5m >> repeat_interval: 24h >> group_by: ['alertname', 'severity'] >> matchers: >> - severity="critical" >> - severity!="resolved" >> >> - receiver: email_warning >> group_interval: 10m >> group_wait: 5m >> repeat_interval: 24h >> group_by: ['alertname', 'severity'] >> matchers: >> - severity="warning" >> - severity!="resolved" >> >> - receiver: email_resolved >> match_re: >> severity: ^(resolved)$ >> >> >> the match_re part from resolved is from good ol' friend chatgpt >> >> Brian Candler schrieb am Samstag, 17. Juni 2023 um 00:48:14 UTC+2: >> >>> You'll need to show the rest of your alertmanager config. "Receivers" by >>> themselves don't do anything; you have to refer to them in routing rules. >>> >>> On Friday, 16 June 2023 at 09:23:56 UTC Kolja Krückmann wrote: >>> >>>> Or do I need to change it to explicitly say in the crit and warn mail >>>> to range over the .Alerts.Firing in order to get the third receiver to >>>> work? >>>> >>>> receivers: >>>> - name: 'email' >>>> email_configs: >>>> - to: '[email protected]' >>>> send_resolved: false >>>> headers: >>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{ >>>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}' >>>> >>>> >>>> - name: 'email_warning' >>>> email_configs: >>>> - to: '[email protected]' >>>> send_resolved: false >>>> headers: >>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ >>>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}' >>>> >>>> >>>> - name: 'email_resolved' >>>> email_configs: >>>> - to: '[email protected]' >>>> send_resolved: true >>>> headers: >>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ >>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}' >>>> >>>> Kolja Krückmann schrieb am Freitag, 16. Juni 2023 um 11:21:59 UTC+2: >>>> >>>>> I changed the config to this: >>>>> >>>>> - name: 'email_resolved' >>>>> email_configs: >>>>> - to: '[email protected]' >>>>> send_resolved: true >>>>> headers: >>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ >>>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}' >>>>> >>>>> But this is sending absolutly nothing. What did I do wrong? >>>>> >>>>> Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2: >>>>> >>>>>> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert >>>>>> group is likely to contain a mixture of firing alerts and resolved >>>>>> alerts. >>>>>> >>>>>> Also, note that your "send_resolved: true" receiver doesn't only send >>>>>> resolved messages: it will send on firing *and* on resolved. Hence the >>>>>> subject "Resolved: ..." is likely to be very misleading. >>>>>> >>>>>> On Friday, 9 June 2023 at 13:42:13 UTC+1 Kolja Krückmann wrote: >>>>>> >>>>>>> Got it working like this: >>>>>>> >>>>>>> receivers: >>>>>>> - name: 'email' >>>>>>> email_configs: >>>>>>> - to: '[email protected]' >>>>>>> send_resolved: false >>>>>>> headers: >>>>>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: >>>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>>> >>>>>>> - name: 'email_warning' >>>>>>> email_configs: >>>>>>> - to: '[email protected]' >>>>>>> send_resolved: false >>>>>>> headers: >>>>>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: >>>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>>> >>>>>>> - name: 'email_resolved' >>>>>>> email_configs: >>>>>>> - to: '[email protected]' >>>>>>> send_resolved: true >>>>>>> headers: >>>>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: >>>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>>> >>>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 >>>>>>> UTC+2: >>>>>>> >>>>>>>> [EDIT] >>>>>>>> Also I want to split the firing and resolved alerts. >>>>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 >>>>>>>> UTC+2: >>>>>>>> >>>>>>>>> Hi all, I tried it now for two days straight and I just don't get >>>>>>>>> it working. >>>>>>>>> >>>>>>>>> First of all where do I actually find the expressions I can use to >>>>>>>>> define the subject under the header? I am still missing a template I >>>>>>>>> can >>>>>>>>> use? I don't really have one in my Prom/Alert-files. >>>>>>>>> Second of all my current receivers are configured like so: >>>>>>>>> >>>>>>>>> receivers: >>>>>>>>> - name: 'email' >>>>>>>>> email_configs: >>>>>>>>> - to: '[email protected]' >>>>>>>>> send_resolved: true >>>>>>>>> headers: >>>>>>>>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{ >>>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>>> >>>>>>>>> Like Brian said, I want to iterate in the second half of that >>>>>>>>> "statement" (I don't actually know how these {{ .xyz }} are called) >>>>>>>>> through >>>>>>>>> all instances and the first one should just say which alert is now >>>>>>>>> firing. >>>>>>>>> I also tried it as followed: >>>>>>>>> >>>>>>>>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{ >>>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>>> and >>>>>>>>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{ >>>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>>> >>>>>>>>> but nothing is really working. >>>>>>>>> >>>>>>>>> All I really want is the Subject like so e.x.: >>>>>>>>> InstanceDown - SVR-DS01 SVR-DC11 >>>>>>>>> ^^^^^^^^ ^^^^^^^^ ^^^^^^^^ >>>>>>>>> Alertname - Instance1 Instance2 (if it is also possible >>>>>>>>> to seperate the instances via comma would be perfect) >>>>>>>>> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2: >>>>>>>>> >>>>>>>>>> As its name implies, "CommonLabels" contains only those labels >>>>>>>>>> which are common to all alerts in the group. If there are multiple >>>>>>>>>> instances, then the "instance" label is not common to all alerts, so >>>>>>>>>> won't >>>>>>>>>> be in this object. >>>>>>>>>> >>>>>>>>>> You want to iterate over "Alerts" (or "Alerts.Firing") and look >>>>>>>>>> at the Labels.instance within each alert. You will find an example >>>>>>>>>> showing >>>>>>>>>> how to do that here: >>>>>>>>>> >>>>>>>>>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts >>>>>>>>>> >>>>>>>>>> Additional references: >>>>>>>>>> https://prometheus.io/docs/alerting/latest/notifications/#data >>>>>>>>>> >>>>>>>>>> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/ >>>>>>>>>> >>>>>>>>>> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote: >>>>>>>>>> >>>>>>>>>>> Currently my alertmanager.yml looks like this: >>>>>>>>>>> >>>>>>>>>>> receivers: >>>>>>>>>>> - name: 'email' >>>>>>>>>>> email_configs: >>>>>>>>>>> - to: '[email protected]' >>>>>>>>>>> send_resolved: true >>>>>>>>>>> headers: >>>>>>>>>>> Subject: '{{ .CommonLabels.alertname }} - {{ >>>>>>>>>>> .CommonLabels.instance }}' >>>>>>>>>>> >>>>>>>>>>> Unfortunately the subject in the mail now only displays both >>>>>>>>>>> labels if only there is an alert for a single instance (like High >>>>>>>>>>> CPU Usage >>>>>>>>>>> on core 0,1,2,3 on one instance (see screenshot attatched)) As soon >>>>>>>>>>> as the >>>>>>>>>>> alert is for more then just one instance the instance lable is >>>>>>>>>>> completly >>>>>>>>>>> missing... Why is it missing? And how can I change it to display >>>>>>>>>>> all >>>>>>>>>>> instances affected by that alert. >>>>>>>>>>> >>>>>>>>>>> Kind regards. >>>>>>>>>>> Kolja >>>>>>>>>>> >>>>>>>>>>> [image: mail.png] >>>>>>>>>>> >>>>>>>>>>> Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 >>>>>>>>>>> UTC+2: >>>>>>>>>>> >>>>>>>>>>> Hi y'all >>>>>>>>>>> I'm trying to change the way the mail from my alertmanger looks >>>>>>>>>>> and behaves. >>>>>>>>>>> My goal is to have the subject list the alertname and the >>>>>>>>>>> targets detected by that alert. (I need it like that for future >>>>>>>>>>> todo's) >>>>>>>>>>> My first problem is, i dont have any templates in my files. Do I >>>>>>>>>>> need to clone one from git? How does the prom know, I have a >>>>>>>>>>> template? >>>>>>>>>>> The second one would be to change the subject in order for my >>>>>>>>>>> requirements to fit. >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> Kolja >>>>>>>>>>> >>>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8e36f83b-03bb-4295-be42-9884391846dfn%40googlegroups.com.
