I checked the json file and the tagging was correct. Here's an example:
{
"labels": {
"cluster": "X Stage Servers",
"env": "dev"
},
"targets": [
"x:9100",
"y:9100",
"z:9100"
]
},
This is being sent to the production/default channel.
On Friday, August 12, 2022 at 11:29:34 AM UTC-4 Brian Candler wrote:
> Firstly, I'd drop the "continue: true" lines. They are not required, and
> are just going to cause confusion.
>
> The 'slack' and 'production' receivers are both sending to #prod-channel.
> So you'll hit this if the env is not exactly "dev". I suggest you look in
> detail at the alerts themselves: maybe they're tagging with "Dev" or "dev "
> (with a hidden space).
>
> If you change the default 'slack' receiver to go to a different channel,
> or use a different title/text template, it will be easier to see if this is
> the problem or not.
>
>
> On Friday, 12 August 2022 at 09:36:22 UTC+1 rs wrote:
>
>> Hi everyone! I am configuring alertmanager to send outputs to a prod
>> slack channel and dev slack channel. I have checked with the routing tree
>> editor and everything should be working correctly.
>> However, I am seeing some (not all) alerts that are tagged with 'env:
>> dev' being sent to the prod slack channel. Is there some sort of old
>> configuration caching happening? Is there a way to flush this out?
>>
>> --- Alertmanager.yml ---
>> global:
>> http_config:
>> proxy_url: 'xyz'
>> templates:
>> - templates/*.tmpl
>> route:
>> group_by: [cluster,alertname]
>> group_wait: 10s
>> group_interval: 30m
>> repeat_interval: 24h
>> receiver: 'slack'
>> routes:
>> - receiver: 'production'
>> match:
>> env: 'prod'
>> continue: true
>> - receiver: 'staging'
>> match:
>> env: 'dev'
>> continue: true
>> receivers:
>> #Fallback option - Default set to production server
>> - name: 'slack'
>> slack_configs:
>> - api_url: 'api url'
>> channel: '#prod-channel'
>> send_resolved: true
>> color: '{{ template "slack.color" . }}'
>> title: '{{ template "slack.title" . }}'
>> text: '{{ template "slack.text" . }}'
>> actions:
>> - type: button
>> text: 'Query :mag:'
>> url: '{{ (index .Alerts 0).GeneratorURL }}'
>> - type: button
>> text: 'Silence :no_bell:'
>> url: '{{ template "__alert_silence_link" . }}'
>> - type: button
>> text: 'Dashboard :grafana:'
>> url: '{{ (index .Alerts 0).Annotations.dashboard }}'
>> - name: 'staging'
>> slack_configs:
>> - api_url: 'api url'
>> channel: '#staging-channel'
>> send_resolved: true
>> color: '{{ template "slack.color" . }}'
>> title: '{{ template "slack.title" . }}'
>> text: '{{ template "slack.text" . }}'
>> actions:
>> - type: button
>> text: 'Query :mag:'
>> url: '{{ (index .Alerts 0).GeneratorURL }}'
>> - type: button
>> text: 'Silence :no_bell:'
>> url: '{{ template "__alert_silence_link" . }}'
>> - type: button
>> text: 'Dashboard :grafana:'
>> url: '{{ (index .Alerts 0).Annotations.dashboard }}'
>> - name: 'production'
>> slack_configs:
>> - api_url: 'api url'
>> channel: '#prod-channel'
>> send_resolved: true
>> color: '{{ template "slack.color" . }}'
>> title: '{{ template "slack.title" . }}'
>> text: '{{ template "slack.text" . }}'
>> actions:
>> - type: button
>> text: 'Query :mag:'
>> url: '{{ (index .Alerts 0).GeneratorURL }}'
>> - type: button
>> text: 'Silence :no_bell:'
>> url: '{{ template "__alert_silence_link" . }}'
>> - type: button
>> text: 'Dashboard :grafana:'
>> url: '{{ (index .Alerts 0).Annotations.dashboard }}'
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/2e6397ba-0e92-42b9-83e1-a919654025ccn%40googlegroups.com.
