I think the part you've missed is that an *alertmanager* webhook message is completely different to a *discord* webhook message. You need an application in between to translate - that is, it receives an alertmanager webhook message, and send it on to discord with a different payload.
If you google "alertmanager discord" then the first two hits are projects which do exactly this: https://github.com/benjojo/alertmanager-discord https://github.com/rogerrum/alertmanager-discord Now, you linked to this bit of helm chart: https://github.com/cbirkenbeul/k3s-gitops/blob/main/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml That doesn't use alertmanager webhooks at all. It uses the *slack* sending capability of alertmanager. Does Discord accept Slack-formatted API messages? If so, then it might work. This suggests that it could: https://stackoverflow.com/questions/66076875/discord-slack-compatible-webhook But there's a separate API endpoint for accepting Slack-format requests: https://discord.com/developers/docs/resources/webhook#execute-slackcompatible-webhook So either: 1. you're not POSTing to the correct endpoint (the Slack-compatible one) 2. something about alertmanager's request makes Discord unhappy (i.e. it's not as Slack-compatible as it claims to be) If you think it's case (2) then you can take it up with Discord support - although they probably will want to see the body of the HTTP request, and the body of the 400 response that comes back. On Tuesday, 30 August 2022 at 14:00:04 UTC+1 Fabrice Semti wrote: > > > *Best regards,* > > *Fabrice Semti* > > 📧 [email protected] 📧 [email protected] > 🌐 fabricesemti.com <https://fabricesemti.com/> > > > ---------- Forwarded message --------- > Feladó: Fabrice Semti <[email protected]> > Date: 2022. aug. 30., K, 13:42 > Subject: Re: [prometheus-users] Re: discord - "unrecoverable error" > To: Brian Candler <[email protected]> > > > Hi Brian, > > Thanks for the reply. > > You haven't said what you've configured as the receiver (target) of the >> webhook. > > > So I have created a webhook on my discord channel. (The same webhook was > tested on the online tester) > > [image: image.png] > > Therefore, if you're pointing the Prometheus alertmanager webhook receiver >> *directly* at a Discord API, it definitely won't work. You will need some >> middleware to translate the Alertmanager webhook payload into something the >> target API understands. Google "alertmanager discord" for some options. >> > > I have based my config on this one > <https://github.com/cbirkenbeul/k3s-gitops/blob/main/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml>, > > which seems to be in line with what I found for "alertmanager discord"... > > My config is: > > alertmanager: > config: > global: > slack_api_url: "${SECRET_ALERT_MANAGER_DISCORD_WEBHOOK}" > resolve_timeout: 5m > receivers: > - name: "null" > - name: "discord" > slack_configs: > - channel: "#prometheus" > icon_url: https://avatars3.githubusercontent.com/u/3380462 > username: "Prometheus" > send_resolved: true > title: |- > [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ > .Alerts.Firing | len }}{{ end }}] {{ if ne .CommonAnnotations.summary > ""}}{{ .CommonAnnotations.summary }}{{ else }}{{ .CommonLabels.alertname > }}{{ end }} > text: >- > {{ range .Alerts -}} > **Alert:** {{ .Annotations.title }}{{ if > .Labels.severity }} - `{{ .Labels.severity }}`{{ end }} > **Description:** {{ if ne .Annotations.description > ""}}{{ .Annotations.description }}{{else}}N/A{{ end }} > **Details:** > {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ > .Value }}` > {{ end }} > {{ end }} > route: > group_by: ["alertname", "job"] > group_wait: 30s > group_interval: 5m > repeat_interval: 6h > receiver: "discord" > routes: > - receiver: "null" > match: > alertname: Watchdog > - receiver: "discord" > match_re: > severity: critical|warning|error > continue: true > inhibit_rules: > - source_match: > severity: "critical" > target_match: > severity: "warning" > equal: ["alertname", "namespace"] > > ingress: > enabled: true > pathType: Prefix > ingressClassName: "nginx" > annotations: > cert-manager.io/cluster-issuer: "letsencrypt-production" > external-dns.alpha.kubernetes.io/target: "ipv4.${SECRET_DOMAIN}" > external-dns/is-public: "true" > hajimari.io/enable: "true" > hajimari.io/icon: "fire-alert" > # nginx.ingress.kubernetes.io/auth-url: "https://oauth. > ${SECRET_DOMAIN}/oauth2/auth" > # nginx.ingress.kubernetes.io/auth-signin: "https://oauth. > ${SECRET_DOMAIN}/oauth2/start?rd=$scheme://$best_http_host$request_uri" > # nginx.ingress.kubernetes.io/auth-response-headers: > "x-auth-request-user, x-auth-request-email, x-auth-request-access-token" > hosts: > - &host "alert-manager.${SECRET_DOMAIN}" > tls: > - hosts: > - *host > secretName: "alert-manager-tls" > alertmanagerSpec: > storage: > volumeClaimTemplate: > spec: > storageClassName: "hdd-nfs-storage" > resources: > requests: > storage: 1Gi > > resources: > requests: > cpu: 11m > memory: 37M > > > > You can of course make one alertmanager receiver which does both, which is >> perhaps what you're already doing: >> >> receivers: >> - name: discord/slack >> slack_configs: >> - ... >> webhook_configs: >> - ... >> >> >> Yes, pretty much that is what I am aiming for, and I only lightly > modified what I have seen in other repos (such as above or this one > <https://github.com/North14/home-cluster/blob/125cc4938947e9ce07df6ba4bc7e3d6170590d41/cluster/apps/monitoring/kube-prometheus-stack/helm-release.yaml> > > ), so I am not sure what is not right... > Any further clues would be much appreciated... > > > > *Best regards,* > > *Fabrice Semti* > > 📧 [email protected] 📧 [email protected] > 🌐 fabricesemti.com <https://fabricesemti.com/> > > > Brian Candler <[email protected]> ezt írta (időpont: 2022. aug. 30., K, > 11:11): > >> You haven't said what you've configured as the receiver (target) of the >> webhook. >> >> Whatever it is, it is responding with a 400 error, which means that the >> request being sent is invalid. >> >> The format of the webhook payload sent by Alertmanager is *fixed*. For >> an example see: >> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config >> >> This will be different to what a "Discord webhook tester" sends. >> >> Therefore, if you're pointing the Prometheus alertmanager webhook >> receiver *directly* at a Discord API, it definitely won't work. You will >> need some middleware to translate the Alertmanager webhook payload into >> something the target API understands. Google "alertmanager discord" for >> some options. >> >> As for sending to slack, you should use the Alertmanager built-in slack >> integration, not webhook: >> https://prometheus.io/docs/alerting/latest/configuration/#slack_config >> >> You can of course make one alertmanager receiver which does both, which >> is perhaps what you're already doing: >> >> receivers: >> - name: discord/slack >> slack_configs: >> - ... >> webhook_configs: >> - ... >> >> On Tuesday, 30 August 2022 at 08:03:42 UTC+1 [email protected] wrote: >> >>> Hi Everyone, >>> >>> Seeking some help here - very new to Prometheus/Alert Manager, must add. >>> >>> - I set up the kube-prometheus-stack as per: >>> https://github.com/fabricesemti80/home-stack/blob/42f23c6e62184b037908e84d52776a717889ff20/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml#L81 >>> >>> - log errors are similar to this: >>> >>> ts=2022-08-30T06:50:59.456Z caller=dispatch.go:354 level=error >>> component=dispatcher msg="Notify for alerts failed" num_alerts=3 >>> err="discord/slack[0]: notify retry canceled due to unrecoverable error >>> after 1 attempts: channel \"#prometheus\": unexpected status code 400: >>> {\"attachments\": [\"0\"]}" >>> >>> - I used this site <https://disforge.com/tool/webhook-tester> to test >>> the webhook >>> >>> - the test - with the same parameters used in the manifest - was >>> successful (see attached screenshot), therefore I think my settings are >>> correct. >>> >>> Any suggestions what am I missing please? >>> >>> Thanks, >>> F. >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Prometheus Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/prometheus-users/uqNf-alj0i0/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/e7f7f14f-514f-474c-8052-5bae2c0a6a64n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/e7f7f14f-514f-474c-8052-5bae2c0a6a64n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/33a1a588-f170-4b26-9285-fdc7db3fd509n%40googlegroups.com.
