On Fri, 3 Dec 2021 at 19:08, Matt Miller <[email protected]> wrote: > Hi, > > In the documentation under the HA section, it mentions: > "It's important not to load balance traffic between Prometheus and its > Alertmanagers, but instead, point Prometheus to a list of all > Alertmanagers." > > I'm curious if this is strictly for high availability and network > partitioning concerns, or if there is a more functional reason that every > Alertmanager member needs to receive the alerts. >
Both are true. > > What prompted this question, is that in our three member HA alertmanager > cluster that we've been sending alerts to via a load balancer (from > multiple prometheus instances), we've observed that alerts stored on each > cluster member may have pretty drastically different endsAt times for a > single given alert (one to two minutes). We believe that this may be > contributing to random flapping alerts, that prometheus indicates has been > firing the entire duration. > This is what can happen if you don't follow the documentation. Alerts are not passed between alertmanagers, you need to send them to all of them. Brian > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/4b4e8a88-1243-4591-b2d7-43bd550738een%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/4b4e8a88-1243-4591-b2d7-43bd550738een%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLrCcajp83XoZ4k6RZ34xXPO%2BYzYmsWEcLjuHYfDVSTxvw%40mail.gmail.com.
