And the URL you've configured, it definitely looks like
/webhooks/{webhook.id}/{webhook.token}/slack
?
On Tuesday, 30 August 2022 at 15:22:03 UTC+1 Fabrice Semti wrote:
> Thanks for your reply.
>
> I mean, if I replace the SECRET_ALERT_MANAGER_DISCORD_WEBHOOK with
> SECRET_ALERT_MANAGER_SLACK_WEBHOOK(so using instead of a Discord webhook an
> actual Slack one - to a freshly created slack channel of mine) the alerts
> are indeed coming in..
>
>
> [image: image.png]
>
> [image: image.png]
> So indeed something must be amiss between alert manager and/or
> Discord...will do some reading / reaching out to Discord support, I thought
> the Discord webhook by default interchangeable with the Slack-one, at least
> the helmreleases I have seen seemed to indicate this.
> Will circle back here if that avenue is exhausted...
>
>
> *Best regards,*
>
> *Fabrice Semti*
>
> 📧 [email protected] 📧 [email protected]
> 🌐 fabricesemti.com <https://fabricesemti.com/>
> Brian Candler <[email protected]> ezt írta (időpont: 2022. aug. 30., K,
> 14:15):
>
>> I think the part you've missed is that an *alertmanager* webhook message
>> is completely different to a *discord* webhook message. You need an
>> application in between to translate - that is, it receives an alertmanager
>> webhook message, and send it on to discord with a different payload.
>>
>> If you google "alertmanager discord" then the first two hits are projects
>> which do exactly this:
>>
>> https://github.com/benjojo/alertmanager-discord
>> https://github.com/rogerrum/alertmanager-discord
>>
>> Now, you linked to this bit of helm chart:
>>
>> https://github.com/cbirkenbeul/k3s-gitops/blob/main/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml
>>
>> That doesn't use alertmanager webhooks at all. It uses the *slack*
>> sending capability of alertmanager.
>>
>> Does Discord accept Slack-formatted API messages? If so, then it might
>> work. This suggests that it could:
>>
>> https://stackoverflow.com/questions/66076875/discord-slack-compatible-webhook
>> But there's a separate API endpoint for accepting Slack-format requests:
>>
>> https://discord.com/developers/docs/resources/webhook#execute-slackcompatible-webhook
>>
>> So either:
>> 1. you're not POSTing to the correct endpoint (the Slack-compatible one)
>> 2. something about alertmanager's request makes Discord unhappy (i.e.
>> it's not as Slack-compatible as it claims to be)
>>
>> If you think it's case (2) then you can take it up with Discord support -
>> although they probably will want to see the body of the HTTP request, and
>> the body of the 400 response that comes back.
>>
>> On Tuesday, 30 August 2022 at 14:00:04 UTC+1 Fabrice Semti wrote:
>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *Fabrice Semti*
>>>
>>> 📧 [email protected] 📧 [email protected]
>>> 🌐 fabricesemti.com <https://fabricesemti.com/>
>>>
>>>
>>> ---------- Forwarded message ---------
>>> Feladó: Fabrice Semti <[email protected]>
>>> Date: 2022. aug. 30., K, 13:42
>>> Subject: Re: [prometheus-users] Re: discord - "unrecoverable error"
>>> To: Brian Candler <[email protected]>
>>>
>>>
>>> Hi Brian,
>>>
>>> Thanks for the reply.
>>>
>>> You haven't said what you've configured as the receiver (target) of the
>>>> webhook.
>>>
>>>
>>> So I have created a webhook on my discord channel. (The same webhook was
>>> tested on the online tester)
>>>
>>> [image: image.png]
>>>
>>> Therefore, if you're pointing the Prometheus alertmanager webhook
>>>> receiver *directly* at a Discord API, it definitely won't work. You will
>>>> need some middleware to translate the Alertmanager webhook payload into
>>>> something the target API understands. Google "alertmanager discord" for
>>>> some options.
>>>>
>>>
>>> I have based my config on this one
>>> <https://github.com/cbirkenbeul/k3s-gitops/blob/main/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml>,
>>>
>>> which seems to be in line with what I found for "alertmanager discord"...
>>>
>>> My config is:
>>>
>>> alertmanager:
>>> config:
>>> global:
>>> slack_api_url: "${SECRET_ALERT_MANAGER_DISCORD_WEBHOOK}"
>>> resolve_timeout: 5m
>>> receivers:
>>> - name: "null"
>>> - name: "discord"
>>> slack_configs:
>>> - channel: "#prometheus"
>>> icon_url:
>>> https://avatars3.githubusercontent.com/u/3380462
>>> username: "Prometheus"
>>> send_resolved: true
>>> title: |-
>>> [{{ .Status | toUpper }}{{ if eq .Status "firing"
>>> }}:{{ .Alerts.Firing | len }}{{ end }}] {{ if ne .CommonAnnotations.summary
>>> ""}}{{ .CommonAnnotations.summary }}{{ else }}{{ .CommonLabels.alertname
>>> }}{{ end }}
>>> text: >-
>>> {{ range .Alerts -}}
>>> **Alert:** {{ .Annotations.title }}{{ if
>>> .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
>>> **Description:** {{ if ne .Annotations.description
>>> ""}}{{ .Annotations.description }}{{else}}N/A{{ end }}
>>> **Details:**
>>> {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{
>>> .Value }}`
>>> {{ end }}
>>> {{ end }}
>>> route:
>>> group_by: ["alertname", "job"]
>>> group_wait: 30s
>>> group_interval: 5m
>>> repeat_interval: 6h
>>> receiver: "discord"
>>> routes:
>>> - receiver: "null"
>>> match:
>>> alertname: Watchdog
>>> - receiver: "discord"
>>> match_re:
>>> severity: critical|warning|error
>>> continue: true
>>> inhibit_rules:
>>> - source_match:
>>> severity: "critical"
>>> target_match:
>>> severity: "warning"
>>> equal: ["alertname", "namespace"]
>>>
>>> ingress:
>>> enabled: true
>>> pathType: Prefix
>>> ingressClassName: "nginx"
>>> annotations:
>>> cert-manager.io/cluster-issuer: "letsencrypt-production"
>>> external-dns.alpha.kubernetes.io/target: "
>>> ipv4.${SECRET_DOMAIN}"
>>> external-dns/is-public: "true"
>>> hajimari.io/enable: "true"
>>> hajimari.io/icon: "fire-alert"
>>> # nginx.ingress.kubernetes.io/auth-url: "https://oauth.
>>> ${SECRET_DOMAIN}/oauth2/auth"
>>> # nginx.ingress.kubernetes.io/auth-signin: "https://oauth.
>>> ${SECRET_DOMAIN}/oauth2/start?rd=$scheme://$best_http_host$request_uri"
>>> # nginx.ingress.kubernetes.io/auth-response-headers:
>>> "x-auth-request-user, x-auth-request-email, x-auth-request-access-token"
>>> hosts:
>>> - &host "alert-manager.${SECRET_DOMAIN}"
>>> tls:
>>> - hosts:
>>> - *host
>>> secretName: "alert-manager-tls"
>>> alertmanagerSpec:
>>> storage:
>>> volumeClaimTemplate:
>>> spec:
>>> storageClassName: "hdd-nfs-storage"
>>> resources:
>>> requests:
>>> storage: 1Gi
>>>
>>> resources:
>>> requests:
>>> cpu: 11m
>>> memory: 37M
>>>
>>>
>>>
>>> You can of course make one alertmanager receiver which does both, which
>>>> is perhaps what you're already doing:
>>>>
>>>> receivers:
>>>> - name: discord/slack
>>>> slack_configs:
>>>> - ...
>>>> webhook_configs:
>>>> - ...
>>>>
>>>>
>>>> Yes, pretty much that is what I am aiming for, and I only lightly
>>> modified what I have seen in other repos (such as above or this one
>>> <https://github.com/North14/home-cluster/blob/125cc4938947e9ce07df6ba4bc7e3d6170590d41/cluster/apps/monitoring/kube-prometheus-stack/helm-release.yaml>
>>>
>>> ), so I am not sure what is not right...
>>> Any further clues would be much appreciated...
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *Fabrice Semti*
>>>
>>> 📧 [email protected] 📧 [email protected]
>>> 🌐 fabricesemti.com <https://fabricesemti.com/>
>>>
>>>
>>> Brian Candler <[email protected]> ezt írta (időpont: 2022. aug. 30., K,
>>> 11:11):
>>>
>>>> You haven't said what you've configured as the receiver (target) of the
>>>> webhook.
>>>>
>>>> Whatever it is, it is responding with a 400 error, which means that the
>>>> request being sent is invalid.
>>>>
>>>> The format of the webhook payload sent by Alertmanager is *fixed*. For
>>>> an example see:
>>>> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
>>>>
>>>> This will be different to what a "Discord webhook tester" sends.
>>>>
>>>> Therefore, if you're pointing the Prometheus alertmanager webhook
>>>> receiver *directly* at a Discord API, it definitely won't work. You will
>>>> need some middleware to translate the Alertmanager webhook payload into
>>>> something the target API understands. Google "alertmanager discord" for
>>>> some options.
>>>>
>>>> As for sending to slack, you should use the Alertmanager built-in slack
>>>> integration, not webhook:
>>>> https://prometheus.io/docs/alerting/latest/configuration/#slack_config
>>>>
>>>> You can of course make one alertmanager receiver which does both, which
>>>> is perhaps what you're already doing:
>>>>
>>>> receivers:
>>>> - name: discord/slack
>>>> slack_configs:
>>>> - ...
>>>> webhook_configs:
>>>> - ...
>>>>
>>>> On Tuesday, 30 August 2022 at 08:03:42 UTC+1 [email protected] wrote:
>>>>
>>>>> Hi Everyone,
>>>>>
>>>>> Seeking some help here - very new to Prometheus/Alert Manager, must
>>>>> add.
>>>>>
>>>>> - I set up the kube-prometheus-stack as per:
>>>>> https://github.com/fabricesemti80/home-stack/blob/42f23c6e62184b037908e84d52776a717889ff20/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml#L81
>>>>>
>>>>> - log errors are similar to this:
>>>>>
>>>>> ts=2022-08-30T06:50:59.456Z caller=dispatch.go:354 level=error
>>>>> component=dispatcher msg="Notify for alerts failed" num_alerts=3
>>>>> err="discord/slack[0]: notify retry canceled due to unrecoverable error
>>>>> after 1 attempts: channel \"#prometheus\": unexpected status code 400:
>>>>> {\"attachments\": [\"0\"]}"
>>>>>
>>>>> - I used this site <https://disforge.com/tool/webhook-tester> to test
>>>>> the webhook
>>>>>
>>>>> - the test - with the same parameters used in the manifest - was
>>>>> successful (see attached screenshot), therefore I think my settings are
>>>>> correct.
>>>>>
>>>>> Any suggestions what am I missing please?
>>>>>
>>>>> Thanks,
>>>>> F.
>>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "Prometheus Users" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/prometheus-users/uqNf-alj0i0/unsubscribe
>>>> .
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/prometheus-users/e7f7f14f-514f-474c-8052-5bae2c0a6a64n%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/prometheus-users/e7f7f14f-514f-474c-8052-5bae2c0a6a64n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Prometheus Users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/prometheus-users/uqNf-alj0i0/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/33a1a588-f170-4b26-9285-fdc7db3fd509n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/33a1a588-f170-4b26-9285-fdc7db3fd509n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/e4768237-b7b0-47d6-96c2-a0bd4e454ed1n%40googlegroups.com.
