Brian, a wildcard is just a gibberish of standard card messages proposed by Google. I believe you're already familiar with https://developers.google.com/chat/api/guides/message-formats/cards#apps-script
Based on Mr-Karan's example (https://github.com/mr-karan/calert/blob/main/docs/examples/send_alert.sh) my assumption is as follows: 1) Calert retrieves an unstructured json alert, which alertmanager puts to host:6000 2) After that calert converts such a raw json to the one suiting it's template 3) It is then posting it to a google chat via curl Hence, calert is capable of processing the message under the hood, converting it to Google API-friendly json format and this is I guess I was looking for, but lack general understanding of whether it is possible to amend среда, 14 июня 2023 г. в 12:21:49 UTC+6, Brian Candler: > > It is basically possible by converting the message in a wildcard > structure acceptable by Google Chat API > > I don't know what you mean by a "wildcard structure". Can you give an > example, and/or point to documentation? Is it using tags within the > message field, or does it require a different JSON structure entirely? > > On Wednesday, 14 June 2023 at 05:12:53 UTC+1 Near N wrote: > >> Correct, that's same calert I was mentioning previously. I understand the >> use case of default template, but thought it is possible to do highlighting >> within the template. It is basically possible by converting the message in >> a wildcard structure acceptable by Google Chat API, but I couldn't find any >> options to do so with calert template, but thanks for suggestions >> >> понедельник, 12 июня 2023 г. в 19:43:10 UTC+6, Brian Candler: >> >>> If we're talking about the same "calert >>> <https://github.com/mr-karan/calert>" then it looks like it already >>> supports user-supplied templating of the message content: >>> >>> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/message.go#L18-L53 >>> >>> The sample config shows how to configure it: >>> https://github.com/mr-karan/calert/blob/main/config.sample.toml#L13 >>> https://github.com/mr-karan/calert/blob/main/static/message.tmpl >>> >>> If you want to do colours and highlighting, it would depend whether the >>> Google Chat API accepts HTML tags, or some other way of marking up the >>> content. You'll need to find that out. >>> >>> The variables available in the templates look to be very similar to >>> those provided in Alertmanager templates: >>> https://prometheus.io/docs/alerting/latest/notifications/ >>> https://prometheus.io/docs/alerting/latest/notification_examples/ >>> >>> In fact, there's a test case which uses exactly the alertmanager code >>> (an instance of alertmgrtmpl.Alert) to drive it: >>> >>> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/google_chat_test.go#L27-L35 >>> >>> On Monday, 12 June 2023 at 11:33:32 UTC+1 Near N wrote: >>> >>>> Brian, thanks for pointing to other integrations. The reason why I was >>>> wondering about curling the messages is that the existing solution >>>> (Calert) >>>> provides little information about templating options. Alerts are incoming, >>>> but with tight information. I'd love to highlight severity in different >>>> colors, add warning icons, e.g. something similar to Zabbix or at least >>>> that can suit basic user-friendly message. >>>> >>>> понедельник, 12 июня 2023 г. в 16:06:34 UTC+6, Brian Candler: >>>> >>>>> I can't speak for the project, but my guess would be it's out of >>>>> scope. There are many existing integrations which use the existing >>>>> webhook >>>>> mechanism: >>>>> >>>>> https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver >>>>> Doing it externally allows the request to be enriched with other data >>>>> which wouldn't be available in a template, and to use a variety of >>>>> outbound >>>>> authentication mechanisms which Alertmanager itself doesn't implement, or >>>>> even non-HTTP protocols like STOMP. >>>>> >>>>> In any case, text templating is a nasty and insecure way to create >>>>> JSON or XML payloads. >>>>> >>>>> > it is possible to converse the receivable JSON on the hop and just >>>>> send notifications via curl method >>>>> >>>>> You'd first have to *receive* the webhook payload from Alertmanager, >>>>> which involves plugging your curl script into a webserver somehow. You >>>>> could probably write one as a CGI script under Apache, for example. >>>>> >>>>> But why would you do that, when you have an existing tested and >>>>> working solution? >>>>> >>>>> On Monday, 12 June 2023 at 08:13:50 UTC+1 Near N wrote: >>>>> >>>>>> Thanks for reply Brian. I do realize, that Google Chat API requires >>>>>> a particular JSON format, but was assuming Alertmanager is capable of >>>>>> doing >>>>>> it by using particular template, which is wrong. Now the questions is >>>>>> whether it is worth creating a PR to create such a feature or it is out >>>>>> of >>>>>> Alertmanager's standard scope? >>>>>> >>>>>> Another question is perhaps it is possible to converse the receivable >>>>>> JSON on the hop and just send notifications via curl method? I mean if >>>>>> the >>>>>> endpoint contains the following alert " >>>>>> [{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":" >>>>>> http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The >>>>>> >>>>>> number of available Cassandra nodes has >>>>>> changed","server":"main","severity":"Critical"}}]", would it be possible >>>>>> to >>>>>> retrieve this json and post it via curl? This, I assume, wouldn't solve >>>>>> my >>>>>> problem, cause I believe Google API expects something peculiar in terms >>>>>> of >>>>>> the message it receives. >>>>>> >>>>>> понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler: >>>>>> >>>>>>> > Invalid JSON payload received. >>>>>>> >>>>>>> Alertmanager's webhook sends a JSON message with a *fixed* payload >>>>>>> format. See >>>>>>> >>>>>>> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config >>>>>>> *"The Alertmanager will send HTTP POST requests in the following >>>>>>> JSON format to the configured endpoint"* >>>>>>> >>>>>>> You can't override it with a template. The Alertmanager JSON format >>>>>>> is different to what Google Chat requires, and therefore you can't send >>>>>>> it >>>>>>> directly. >>>>>>> >>>>>>> That's why you need a separate piece of software to do the >>>>>>> conversion (Calert in your case) - it receives the Alertmanager webhook >>>>>>> JSON payload, builds a new payload for the target, and then sends it >>>>>>> there. >>>>>>> >>>>>>> On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote: >>>>>>> >>>>>>>> Good day. I was wondering if there is a way to configure >>>>>>>> alertmanager to send notifications to google chat via a webhook. >>>>>>>> >>>>>>>> The current setup that I use consists of vmalert (loaded with set >>>>>>>> of promQL compatible rules) >> Alertmanager (to group them up) and >>>>>>>> Calert >>>>>>>> (to send them to google chat). Despite the flowchart is working fine, >>>>>>>> I >>>>>>>> found it difficult to amend proposed message template, so decided to >>>>>>>> simplify it by configuring alertmanager to send the alerts. It dawns >>>>>>>> on me >>>>>>>> that it can operate, but I seem to have failed to fine-tune it. Any >>>>>>>> help >>>>>>>> would be appreciated. The alertmanager.yml is below: >>>>>>>> >>>>>>>> global: >>>>>>>> http_config: >>>>>>>> proxy_url: '{{ alertmanager_proxy_url }}' >>>>>>>> >>>>>>>> route: >>>>>>>> receiver: 'my_alerts' >>>>>>>> group_wait: 30s >>>>>>>> group_interval: 60s >>>>>>>> repeat_interval: 15m >>>>>>>> group_by: ['room', 'alertName'] >>>>>>>> >>>>>>>> receivers: >>>>>>>> - name: 'my_alerts' >>>>>>>> webhook_configs: >>>>>>>> - url: '{{ alertmanager_chat_webhook }}' >>>>>>>> send_resolved: true >>>>>>>> >>>>>>>> templates: >>>>>>>> - '{{ alertmanager_working_directory }}/google-chat.tmpl' >>>>>>>> >>>>>>>> >>>>>>>> And here is the error of alertmanager trying to send an alert: >>>>>>>> >>>>>>>> Jun 12 05:26:00 app1-cp alertmanager[140337]: >>>>>>>> ts=2023-06-12T05:26:00.599Z caller=dispatch.go:352 level=error >>>>>>>> component=dispatcher msg="Notify for alerts failed" num_alerts=3 >>>>>>>> err="my_alerts/webhook[0]: notify retry canceled due to unrecoverable >>>>>>>> error >>>>>>>> after 1 attempts: unexpected status code 400: >>>>>>>> https://chat.googleapis.com/v1/spaces/webook_token: {\n >>>>>>>> \"error\": {\n \"code\": 400,\n \"message\": \"Invalid JSON >>>>>>>> payload >>>>>>>> received. Unknown name \\\"receiver\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid JSON payload received. Unknown name \\\"status\\\" at >>>>>>>> 'message': Cannot find field.\\nInvalid JSON payload received. Unknown >>>>>>>> name >>>>>>>> \\\"alerts\\\" at 'message': Cannot find field.\\nInvalid JSON payload >>>>>>>> received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>>>> \\\"commonLabels\\\" >>>>>>>> at 'message': Cannot find field.\\nInvalid JSON payload received. >>>>>>>> Unknown >>>>>>>> name \\\"commonAnnotations\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid >>>>>>>> JSON payload received. Unknown name \\\"externalURL\\\" at 'message': >>>>>>>> Cannot find field.\\nInvalid JSON payload received. Unknown name >>>>>>>> \\\"version\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>>>> payload >>>>>>>> received. Unknown name \\\"groupKey\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>>>> \\\"truncatedAlerts\\\" at 'message': Cannot find field.\",\n >>>>>>>> \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n >>>>>>>> >>>>>>>> \"@type\": \"type.googleapis.com/google.rpc.BadRequest\ >>>>>>>> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n >>>>>>>> \"fieldViolations\": [\n {\n \"field\": >>>>>>>> \"message\",\n >>>>>>>> \"description\": \"Invalid JSON payload received. Unknown >>>>>>>> name >>>>>>>> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>> >>>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>>> \"Invalid JSON payload received. Unknown name \\\"status\\\" at >>>>>>>> 'message': >>>>>>>> Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": >>>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>>> received. >>>>>>>> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n >>>>>>>> >>>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>> >>>>>>>> {\n \"field\": \"message\",\n >>>>>>>> \"description\": >>>>>>>> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at >>>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>>> payload received. Unknown name \\\"commonAnnotations\\\" at 'message': >>>>>>>> Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": >>>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>>> received. >>>>>>>> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n >>>>>>>> >>>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>>> \\\"version\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>> >>>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>>> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at >>>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>>>>> Cannot >>>>>>>> find field.\"\n }\n ]\n }\n ]\n }\n}\n" >>>>>>>> >>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c4d7d5a4-25e9-4c7f-9b23-4c568163732bn%40googlegroups.com.
