Hi Brian, thanks here are the routes to the receivers:
routes:
- receiver: email_critical
group_interval: 1m
group_wait: 5m
repeat_interval: 24h
group_by: ['alertname', 'severity']
matchers:
- severity="critical"
- severity!="resolved"
- receiver: email_warning
group_interval: 10m
group_wait: 5m
repeat_interval: 24h
group_by: ['alertname', 'severity']
matchers:
- severity="warning"
- severity!="resolved"
- receiver: email_resolved
match_re:
severity: ^(resolved)$
the match_re part from resolved is from good ol' friend chatgpt
Brian Candler schrieb am Samstag, 17. Juni 2023 um 00:48:14 UTC+2:
> You'll need to show the rest of your alertmanager config. "Receivers" by
> themselves don't do anything; you have to refer to them in routing rules.
>
> On Friday, 16 June 2023 at 09:23:56 UTC Kolja Krückmann wrote:
>
>> Or do I need to change it to explicitly say in the crit and warn mail to
>> range over the .Alerts.Firing in order to get the third receiver to work?
>>
>> receivers:
>> - name: 'email'
>> email_configs:
>> - to: '[email protected]'
>> send_resolved: false
>> headers:
>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
>>
>>
>> - name: 'email_warning'
>> email_configs:
>> - to: '[email protected]'
>> send_resolved: false
>> headers:
>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
>>
>>
>> - name: 'email_resolved'
>> email_configs:
>> - to: '[email protected]'
>> send_resolved: true
>> headers:
>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
>>
>> Kolja Krückmann schrieb am Freitag, 16. Juni 2023 um 11:21:59 UTC+2:
>>
>>> I changed the config to this:
>>>
>>> - name: 'email_resolved'
>>> email_configs:
>>> - to: '[email protected]'
>>> send_resolved: true
>>> headers:
>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
>>>
>>> But this is sending absolutly nothing. What did I do wrong?
>>>
>>> Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2:
>>>
>>>> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert
>>>> group is likely to contain a mixture of firing alerts and resolved alerts.
>>>>
>>>> Also, note that your "send_resolved: true" receiver doesn't only send
>>>> resolved messages: it will send on firing *and* on resolved. Hence the
>>>> subject "Resolved: ..." is likely to be very misleading.
>>>>
>>>> On Friday, 9 June 2023 at 13:42:13 UTC+1 Kolja Krückmann wrote:
>>>>
>>>>> Got it working like this:
>>>>>
>>>>> receivers:
>>>>> - name: 'email'
>>>>> email_configs:
>>>>> - to: '[email protected]'
>>>>> send_resolved: false
>>>>> headers:
>>>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
>>>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>>>
>>>>> - name: 'email_warning'
>>>>> email_configs:
>>>>> - to: '[email protected]'
>>>>> send_resolved: false
>>>>> headers:
>>>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{
>>>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>>>
>>>>> - name: 'email_resolved'
>>>>> email_configs:
>>>>> - to: '[email protected]'
>>>>> send_resolved: true
>>>>> headers:
>>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>>>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>>>
>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 UTC+2:
>>>>>
>>>>>> [EDIT]
>>>>>> Also I want to split the firing and resolved alerts.
>>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 UTC+2:
>>>>>>
>>>>>>> Hi all, I tried it now for two days straight and I just don't get it
>>>>>>> working.
>>>>>>>
>>>>>>> First of all where do I actually find the expressions I can use to
>>>>>>> define the subject under the header? I am still missing a template I
>>>>>>> can
>>>>>>> use? I don't really have one in my Prom/Alert-files.
>>>>>>> Second of all my current receivers are configured like so:
>>>>>>>
>>>>>>> receivers:
>>>>>>> - name: 'email'
>>>>>>> email_configs:
>>>>>>> - to: '[email protected]'
>>>>>>> send_resolved: true
>>>>>>> headers:
>>>>>>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{
>>>>>>> .Alerts.instance }} {{ end }}"
>>>>>>>
>>>>>>> Like Brian said, I want to iterate in the second half of that
>>>>>>> "statement" (I don't actually know how these {{ .xyz }} are called)
>>>>>>> through
>>>>>>> all instances and the first one should just say which alert is now
>>>>>>> firing.
>>>>>>> I also tried it as followed:
>>>>>>>
>>>>>>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{
>>>>>>> .Alerts.instance }} {{ end }}"
>>>>>>> and
>>>>>>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{
>>>>>>> .Alerts.instance }} {{ end }}"
>>>>>>>
>>>>>>> but nothing is really working.
>>>>>>>
>>>>>>> All I really want is the Subject like so e.x.:
>>>>>>> InstanceDown - SVR-DS01 SVR-DC11
>>>>>>> ^^^^^^^^ ^^^^^^^^ ^^^^^^^^
>>>>>>> Alertname - Instance1 Instance2 (if it is also possible to
>>>>>>> seperate the instances via comma would be perfect)
>>>>>>> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2:
>>>>>>>
>>>>>>>> As its name implies, "CommonLabels" contains only those labels
>>>>>>>> which are common to all alerts in the group. If there are multiple
>>>>>>>> instances, then the "instance" label is not common to all alerts, so
>>>>>>>> won't
>>>>>>>> be in this object.
>>>>>>>>
>>>>>>>> You want to iterate over "Alerts" (or "Alerts.Firing") and look at
>>>>>>>> the Labels.instance within each alert. You will find an example
>>>>>>>> showing how
>>>>>>>> to do that here:
>>>>>>>>
>>>>>>>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts
>>>>>>>>
>>>>>>>> Additional references:
>>>>>>>> https://prometheus.io/docs/alerting/latest/notifications/#data
>>>>>>>>
>>>>>>>> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
>>>>>>>>
>>>>>>>> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote:
>>>>>>>>
>>>>>>>>> Currently my alertmanager.yml looks like this:
>>>>>>>>>
>>>>>>>>> receivers:
>>>>>>>>> - name: 'email'
>>>>>>>>> email_configs:
>>>>>>>>> - to: '[email protected]'
>>>>>>>>> send_resolved: true
>>>>>>>>> headers:
>>>>>>>>> Subject: '{{ .CommonLabels.alertname }} - {{
>>>>>>>>> .CommonLabels.instance }}'
>>>>>>>>>
>>>>>>>>> Unfortunately the subject in the mail now only displays both
>>>>>>>>> labels if only there is an alert for a single instance (like High CPU
>>>>>>>>> Usage
>>>>>>>>> on core 0,1,2,3 on one instance (see screenshot attatched)) As soon
>>>>>>>>> as the
>>>>>>>>> alert is for more then just one instance the instance lable is
>>>>>>>>> completly
>>>>>>>>> missing... Why is it missing? And how can I change it to display all
>>>>>>>>> instances affected by that alert.
>>>>>>>>>
>>>>>>>>> Kind regards.
>>>>>>>>> Kolja
>>>>>>>>>
>>>>>>>>> [image: mail.png]
>>>>>>>>>
>>>>>>>>> Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 UTC+2:
>>>>>>>>>
>>>>>>>>> Hi y'all
>>>>>>>>> I'm trying to change the way the mail from my alertmanger looks
>>>>>>>>> and behaves.
>>>>>>>>> My goal is to have the subject list the alertname and the targets
>>>>>>>>> detected by that alert. (I need it like that for future todo's)
>>>>>>>>> My first problem is, i dont have any templates in my files. Do I
>>>>>>>>> need to clone one from git? How does the prom know, I have a
>>>>>>>>> template?
>>>>>>>>> The second one would be to change the subject in order for my
>>>>>>>>> requirements to fit.
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>> Kolja
>>>>>>>>>
>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/08779d7d-d75e-45f2-bedf-0cbb7950fb82n%40googlegroups.com.
