Good day. I was wondering if there is a way to configure alertmanager to
send notifications to google chat via a webhook.
The current setup that I use consists of vmalert (loaded with set of promQL
compatible rules) >> Alertmanager (to group them up) and Calert (to send
them to google chat). Despite the flowchart is working fine, I found it
difficult to amend proposed message template, so decided to simplify it by
configuring alertmanager to send the alerts. It dawns on me that it can
operate, but I seem to have failed to fine-tune it. Any help would be
appreciated. The alertmanager.yml is below:
global:
http_config:
proxy_url: '{{ alertmanager_proxy_url }}'
route:
receiver: 'my_alerts'
group_wait: 30s
group_interval: 60s
repeat_interval: 15m
group_by: ['room', 'alertName']
receivers:
- name: 'my_alerts'
webhook_configs:
- url: '{{ alertmanager_chat_webhook }}'
send_resolved: true
templates:
- '{{ alertmanager_working_directory }}/google-chat.tmpl'
And here is the error of alertmanager trying to send an alert:
Jun 12 05:26:00 app1-cp alertmanager[140337]: ts=2023-06-12T05:26:00.599Z
caller=dispatch.go:352 level=error component=dispatcher msg="Notify for
alerts failed" num_alerts=3 err="my_alerts/webhook[0]: notify retry
canceled due to unrecoverable error after 1 attempts: unexpected status
code 400: https://chat.googleapis.com/v1/spaces/webook_token: {\n
\"error\": {\n \"code\": 400,\n \"message\": \"Invalid JSON payload
received. Unknown name \\\"receiver\\\" at 'message': Cannot find
field.\\nInvalid JSON payload received. Unknown name \\\"status\\\" at
'message': Cannot find field.\\nInvalid JSON payload received. Unknown name
\\\"alerts\\\" at 'message': Cannot find field.\\nInvalid JSON payload
received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find
field.\\nInvalid JSON payload received. Unknown name \\\"commonLabels\\\"
at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown
name \\\"commonAnnotations\\\" at 'message': Cannot find field.\\nInvalid
JSON payload received. Unknown name \\\"externalURL\\\" at 'message':
Cannot find field.\\nInvalid JSON payload received. Unknown name
\\\"version\\\" at 'message': Cannot find field.\\nInvalid JSON payload
received. Unknown name \\\"groupKey\\\" at 'message': Cannot find
field.\\nInvalid JSON payload received. Unknown name
\\\"truncatedAlerts\\\" at 'message': Cannot find field.\",\n
\"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n
\"@type\": \"type.googleapis.com/google.rpc.BadRequest\",\n
\"fieldViolations\": [\n {\n \"field\": \"message\",\n
\"description\": \"Invalid JSON payload received. Unknown name
\\\"receiver\\\" at 'message': Cannot find field.\"\n },\n
{\n \"field\": \"message\",\n \"description\":
\"Invalid JSON payload received. Unknown name \\\"status\\\" at 'message':
Cannot find field.\"\n },\n {\n \"field\":
\"message\",\n \"description\": \"Invalid JSON payload received.
Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n
},\n {\n \"field\": \"message\",\n
\"description\": \"Invalid JSON payload received. Unknown name
\\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n
{\n \"field\": \"message\",\n \"description\":
\"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at
'message': Cannot find field.\"\n },\n {\n
\"field\": \"message\",\n \"description\": \"Invalid JSON
payload received. Unknown name \\\"commonAnnotations\\\" at 'message':
Cannot find field.\"\n },\n {\n \"field\":
\"message\",\n \"description\": \"Invalid JSON payload received.
Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n
},\n {\n \"field\": \"message\",\n
\"description\": \"Invalid JSON payload received. Unknown name
\\\"version\\\" at 'message': Cannot find field.\"\n },\n
{\n \"field\": \"message\",\n \"description\":
\"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at
'message': Cannot find field.\"\n },\n {\n
\"field\": \"message\",\n \"description\": \"Invalid JSON
payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': Cannot
find field.\"\n }\n ]\n }\n ]\n }\n}\n"
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/d69bb689-66d5-4ce1-94ce-d5c4931aa7cbn%40googlegroups.com.
