Do you have a replica label (in external_labels in Prometheus) that distinguishes the two replicas so that the alerts no longer look the same to Alertmanager?
In that case, you would have to drop that first, see "Removing HA Replica Labels from Alerts" under https://training.promlabs.com/training/relabeling/writing-relabeling-rules/keeping-and-dropping-labels If that is not the problem, maybe your AM instances aren't talking to each other correctly. If you create a silence in one of the AM replicas, does it appear in the other? There should also be log messages about the peer discovery, as well as on the /status page of Alertmanager. On Tue, Mar 23, 2021 at 11:11 PM Nolan Crooks <[email protected]> wrote: > I am also having this issue. I am running identical instances of > Prometheus on the same version though. > On Friday, November 6, 2020 at 8:02:31 AM UTC-8 [email protected] > wrote: > >> Hi. I have a HA Prometheus setup, with 2 instances(x.x.x.x and y.y.y.y) >> scraping exactly the same targets. On the respective machines, Alertmanager >> is also running and a mesh is created. But I am observing that all the >> alerts are getting duplicated and I am receiving every alert twice. >> Alertmanager Version - 0.21.0. >> /usr/local/bin/alertmanager --config.file >> /etc/alertmanager/alertmanager.yml --storage.path /mnt/vol2/alertmanager >> --data.retention=120h --log.level=debug --web.listen-address=x.x.x.x:9093 >> --cluster.listen-address=x.x.x.x:9094 --cluster.peer=y.y.y.y:9094 >> >> Oh, one thing that just popped into my head, for the temporary testing >> period I am running different versions of Prometheus in the instances. >> 2.12.0 in one and 2.20.1 in the other one. Could this also cause this? >> >> Thanks in advance! >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c8245b99-376b-44c2-9a6f-433912a294den%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c8245b99-376b-44c2-9a6f-433912a294den%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5yj7xmdpQ6fP%3DJZ%3DnjCadE3aN0B%3D96zFqN8dYHzK%2BrmQw%40mail.gmail.com.
