Thanks for your replies, guys. We have two replicated prometheus instances scraping the same metrics and sending the same alerts in parallel to alertmanager. We add a label to alerts indicating which prometheus instance the alert is fired from, so that if one prometheus instance is going bad we can silence alerts from that instance. The pain point is that the alert email (grouped) body is bloated with duplicated alert texts from both instances with only one label being different.
We would like to keep the label so that we can silence alerts at prometheus instance level. So, label replacement or aggregating labels in prometheus doesn't seem the right way for us. I think it would work for us if alertmanager can be configured to ignore or collapse certain labels in email texts, like: >From prometheus instance 1: label_A = value1 >From prometheus instance 2: label_A = value2 In alert email: label_A = value1, value2 Regards, Ping On Thursday, June 4, 2020 at 4:07:32 PM UTC-4, Christian Hoffmann wrote: > > Hi, > > On 6/4/20 7:48 PM, '[email protected] <javascript:>' via Prometheus > Users wrote: > > We get the same alert multiple times in the same email, because the > > monitor label (prometheus instance) being different for our simple > > replicated setup. Would be nice to be able to ignore certain labels so > > that alert bodies are higher signal. > > This sounds like it could be done on the Prometheus side using existing > (standard) features. > Any reason why just aggregating away the unwanted label would not work? > > E.g. > > avg without(instance) (some_metric) > > (Depending on the value, other aggregation functions such as sum, min or > max might make more sense) > > Kind regards, > Christian > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c878c840-06c4-4c92-85ac-6ebb63332dbfo%40googlegroups.com.
