Thanks for reply Brian. I do realize, that Google Chat API requires a
particular JSON format, but was assuming Alertmanager is capable of doing
it by using particular template, which is wrong. Now the questions is
whether it is worth creating a PR to create such a feature or it is out of
Alertmanager's standard scope?
Another question is perhaps it is possible to converse the receivable JSON
on the hop and just send notifications via curl method? I mean if the
endpoint contains the following alert
"[{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":"http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The
number of available Cassandra nodes has
changed","server":"main","severity":"Critical"}}]", would it be possible to
retrieve this json and post it via curl? This, I assume, wouldn't solve my
problem, cause I believe Google API expects something peculiar in terms of
the message it receives.
понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler:
> > Invalid JSON payload received.
>
> Alertmanager's webhook sends a JSON message with a *fixed* payload format.
> See
> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
> *"The Alertmanager will send HTTP POST requests in the following JSON
> format to the configured endpoint"*
>
> You can't override it with a template. The Alertmanager JSON format is
> different to what Google Chat requires, and therefore you can't send it
> directly.
>
> That's why you need a separate piece of software to do the conversion
> (Calert in your case) - it receives the Alertmanager webhook JSON payload,
> builds a new payload for the target, and then sends it there.
>
> On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote:
>
>> Good day. I was wondering if there is a way to configure alertmanager to
>> send notifications to google chat via a webhook.
>>
>> The current setup that I use consists of vmalert (loaded with set of
>> promQL compatible rules) >> Alertmanager (to group them up) and Calert (to
>> send them to google chat). Despite the flowchart is working fine, I found
>> it difficult to amend proposed message template, so decided to simplify it
>> by configuring alertmanager to send the alerts. It dawns on me that it can
>> operate, but I seem to have failed to fine-tune it. Any help would be
>> appreciated. The alertmanager.yml is below:
>>
>> global:
>> http_config:
>> proxy_url: '{{ alertmanager_proxy_url }}'
>>
>> route:
>> receiver: 'my_alerts'
>> group_wait: 30s
>> group_interval: 60s
>> repeat_interval: 15m
>> group_by: ['room', 'alertName']
>>
>> receivers:
>> - name: 'my_alerts'
>> webhook_configs:
>> - url: '{{ alertmanager_chat_webhook }}'
>> send_resolved: true
>>
>> templates:
>> - '{{ alertmanager_working_directory }}/google-chat.tmpl'
>>
>>
>> And here is the error of alertmanager trying to send an alert:
>>
>> Jun 12 05:26:00 app1-cp alertmanager[140337]: ts=2023-06-12T05:26:00.599Z
>> caller=dispatch.go:352 level=error component=dispatcher msg="Notify for
>> alerts failed" num_alerts=3 err="my_alerts/webhook[0]: notify retry
>> canceled due to unrecoverable error after 1 attempts: unexpected status
>> code 400: https://chat.googleapis.com/v1/spaces/webook_token: {\n
>> \"error\": {\n \"code\": 400,\n \"message\": \"Invalid JSON payload
>> received. Unknown name \\\"receiver\\\" at 'message': Cannot find
>> field.\\nInvalid JSON payload received. Unknown name \\\"status\\\" at
>> 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name
>> \\\"alerts\\\" at 'message': Cannot find field.\\nInvalid JSON payload
>> received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find
>> field.\\nInvalid JSON payload received. Unknown name \\\"commonLabels\\\"
>> at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown
>> name \\\"commonAnnotations\\\" at 'message': Cannot find field.\\nInvalid
>> JSON payload received. Unknown name \\\"externalURL\\\" at 'message':
>> Cannot find field.\\nInvalid JSON payload received. Unknown name
>> \\\"version\\\" at 'message': Cannot find field.\\nInvalid JSON payload
>> received. Unknown name \\\"groupKey\\\" at 'message': Cannot find
>> field.\\nInvalid JSON payload received. Unknown name
>> \\\"truncatedAlerts\\\" at 'message': Cannot find field.\",\n
>> \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n
>> \"@type\": \"type.googleapis.com/google.rpc.BadRequest\
>> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n
>> \"fieldViolations\": [\n {\n \"field\": \"message\",\n
>> \"description\": \"Invalid JSON payload received. Unknown name
>> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n
>> {\n \"field\": \"message\",\n \"description\":
>> \"Invalid JSON payload received. Unknown name \\\"status\\\" at 'message':
>> Cannot find field.\"\n },\n {\n \"field\":
>> \"message\",\n \"description\": \"Invalid JSON payload received.
>> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n
>> },\n {\n \"field\": \"message\",\n
>> \"description\": \"Invalid JSON payload received. Unknown name
>> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n
>> {\n \"field\": \"message\",\n \"description\":
>> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at
>> 'message': Cannot find field.\"\n },\n {\n
>> \"field\": \"message\",\n \"description\": \"Invalid JSON
>> payload received. Unknown name \\\"commonAnnotations\\\" at 'message':
>> Cannot find field.\"\n },\n {\n \"field\":
>> \"message\",\n \"description\": \"Invalid JSON payload received.
>> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n
>> },\n {\n \"field\": \"message\",\n
>> \"description\": \"Invalid JSON payload received. Unknown name
>> \\\"version\\\" at 'message': Cannot find field.\"\n },\n
>> {\n \"field\": \"message\",\n \"description\":
>> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at
>> 'message': Cannot find field.\"\n },\n {\n
>> \"field\": \"message\",\n \"description\": \"Invalid JSON
>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': Cannot
>> find field.\"\n }\n ]\n }\n ]\n }\n}\n"
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/b5b07cbb-5a82-4503-ac9a-c0dacefd2be3n%40googlegroups.com.
