Unfortunately you cannot have it both ways: either Alertmanager knows about separate alert instances that can be silenced separately, or it doesn't.
I would try to eliminate the need to silence by Prometheus, for example by making the alert expressions resistant to gaps in the data. /MR On Fri, Jun 5, 2020, 05:40 '[email protected]' via Prometheus Users < [email protected]> wrote: > Thanks for your replies, guys. We have two replicated prometheus instances > scraping the same metrics and sending the same alerts in parallel to > alertmanager. We add a label to alerts indicating which prometheus instance > the alert is fired from, so that if one prometheus instance is going bad we > can silence alerts from that instance. The pain point is that the alert > email (grouped) body is bloated with duplicated alert texts from both > instances with only one label being different. > > We would like to keep the label so that we can silence alerts at > prometheus instance level. So, label replacement or aggregating labels in > prometheus doesn't seem the right way for us. I think it would work for us > if alertmanager can be configured to ignore or collapse certain labels in > email texts, like: > > From prometheus instance 1: > label_A = value1 > > From prometheus instance 2: > label_A = value2 > > In alert email: > label_A = value1, value2 > > Regards, > > Ping > > On Thursday, June 4, 2020 at 4:07:32 PM UTC-4, Christian Hoffmann wrote: >> >> Hi, >> >> On 6/4/20 7:48 PM, '[email protected]' via Prometheus Users wrote: >> > We get the same alert multiple times in the same email, because the >> > monitor label (prometheus instance) being different for our simple >> > replicated setup. Would be nice to be able to ignore certain labels so >> > that alert bodies are higher signal. >> >> This sounds like it could be done on the Prometheus side using existing >> (standard) features. >> Any reason why just aggregating away the unwanted label would not work? >> >> E.g. >> >> avg without(instance) (some_metric) >> >> (Depending on the value, other aggregation functions such as sum, min or >> max might make more sense) >> >> Kind regards, >> Christian >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c878c840-06c4-4c92-85ac-6ebb63332dbfo%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c878c840-06c4-4c92-85ac-6ebb63332dbfo%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbGTMg5f-OZjVYha8%2BDy4o1aoOZWKfKdWe%3DAvC%2ByU5d8A%40mail.gmail.com.
