On 18.08.20 00:05, [email protected] wrote: > > routes: > - receiver: 'slack_primary' > group_by: [...] # disables grouping > [...] > > One of the expressions that failed was this one: > > envoy_cluster_health_check_healthy{envoy_cluster_name=~"name1|name2|name3"} == > 0 > > We basically had several alerts from the expression above boiling down to > those > two: > > envoy_cluster_health_check_healthy{envoy_cluster_name=<name>, environment= > "staging" } > envoy_cluster_health_check_healthy{envoy_cluster_name=<name>, environment= > "production" } > > But in slack we got reported with just the "production" one because the two > alerts were clustered and the template didn't take in account that. > > We have currently split the two alerts as follows: > > envoy_cluster_health_check_healthy{envoy_cluster_name=~"name1|name2|name3", > environment="staging"} == 0 > > envoy_cluster_health_check_healthy{envoy_cluster_name=~"name1|name2|name3", > environment="production"} == 0 > > Summing up, is that behaviour expected and we should absolutely change the > templates and/or split the rules on environment?
Interesting... I pretty sure it shouldn't matter if the alerts come from the same rule or from different rules. I'm wondering if this is a bug in Alertmanager. I could imagine that AM only ever thinks about grouping together alerts coming from different rules but never considers breaking up alerts coming from the same rule. Or perhaps this weird behavior only happens with the `[...]` grouping. Just to make sure this hasn't been fixed in the latest release, could you repreduce the behavior in v0.21.0 ? If it still happens, I think you should file a bug in https://github.com/prometheus/alertmanager . -- Björn Rabenstein [PGP-ID] 0x851C3DA17D748D03 [email] [email protected] -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200827161815.GL2356%40jahnn.
