Hello, Doesn't it mean that your alert expression is not reliable enough? maybe you are trying to solve an XY problem here. did you try to make an expression that does not trigger without enough history?
On Mon, Jul 20, 2020 at 10:26 AM 'Holger Freyther' via Prometheus Users < [email protected]> wrote: > tl;dr: How to turn-up new Prometheus instances and not send (or silence) > false alerts? > > My HA setup consists out of running several independent instances that > scrape, evaluate the recording rules and alert. That should be pretty > standard and works great until I need to increase the retention size (or > turn-up in a new cluster, maintenance, ..). To upsize I get compute and new > storage and spawn a new instance of Prometheus. > > My troubles start with alerts using larger windows on increase/rate (e.g. > check progress of a job over a longer period of time). The new instance has > an empty TSDB and just started to scrape it is likely to trigger this alert. > > I wondered if I can silence but AFAIK we make the label sets match across > all instances of the HA set-up to allow Alertmanager to dedup. > > Is there a best practice for this? > > holger > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c9d8c05d-ec07-4adc-b4bf-10cd88b0b5cen%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c9d8c05d-ec07-4adc-b4bf-10cd88b0b5cen%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CALnV8WhBPbmX2TzzDy6i_dX%3DbS0Z0Vmz%2BkqzNWojgh%3DZ28rUKA%40mail.gmail.com.
