Thanks Brian, >From all of the documentation I found, it was unclear that alerts were not also gossiped, thanks for the clarification.
On Friday, December 3, 2021 at 4:22:52 PM UTC-5 Brian Brazil wrote: > On Fri, 3 Dec 2021 at 19:08, Matt Miller <[email protected]> wrote: > >> Hi, >> >> In the documentation under the HA section, it mentions: >> "It's important not to load balance traffic between Prometheus and its >> Alertmanagers, but instead, point Prometheus to a list of all >> Alertmanagers." >> >> I'm curious if this is strictly for high availability and network >> partitioning concerns, or if there is a more functional reason that every >> Alertmanager member needs to receive the alerts. >> > > Both are true. > > >> >> What prompted this question, is that in our three member HA alertmanager >> cluster that we've been sending alerts to via a load balancer (from >> multiple prometheus instances), we've observed that alerts stored on each >> cluster member may have pretty drastically different endsAt times for a >> single given alert (one to two minutes). We believe that this may be >> contributing to random flapping alerts, that prometheus indicates has been >> firing the entire duration. >> > > This is what can happen if you don't follow the documentation. Alerts are > not passed between alertmanagers, you need to send them to all of them. > > Brian > > >> >> Thanks. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/4b4e8a88-1243-4591-b2d7-43bd550738een%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/4b4e8a88-1243-4591-b2d7-43bd550738een%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5ccf42df-79a7-4e2f-82a4-fc6a98d7b406n%40googlegroups.com.
