Hello All, We are using a highly available Prometheus setup by using Prometheus Operator as our driving agent. We have 2 Prometheus instances running along with 2 Alertmanagers configured to receive Alarms. The issue we are observing is, Alertmanager is sending duplicate Clear alert notifications for certain Alarms which is creating discrepancies at the end user. We are using Netcool as our Webhook receiver for Alerts notification.
We are using kube-prometheus-stack helm chart to deploy our Prometheus and alertmanager instances and other components Kube-prometheus-stack helm chart Version: "18.0.1" K8s Version: "v1.20.7" *Below is our Alertmanager configuration used with Netcool:* global: resolve_timeout: 5m receivers: - name: default-receiver webhook_configs: - url: http://<ip>:port/probe/webhook/prometheus <http://10.45.25.136:6180/probe/webhook/prometheus> send_resolved: true - name: Netcool-Receiver webhook_configs: - url: http://<ip>:port/probe/webhook/prometheus <http://10.45.9.73:6180/probe/webhook/prometheus> send_resolved: true route: group_interval: 5m group_wait: 30s receiver: default-receiver repeat_interval: 3h group_by: [ 'alertname' ] routes: - receiver: default-receiver group_wait: 30s group_interval: 5m repeat_interval: 3h continue: true - receiver: Netcool-Receiver group_wait: 10s group_interval: 5m repeat_interval: 3h continue: true templates: - /etc/alertmanager/config/*.tmpl *Below is our Prometheus ecosystem:* [cloud-user@incredibles1-bastion-1 ~]$ kubectl get pods -n occne-infra | grep kube alertmanager-occne-kube-prom-stack-kube-alertmanager-0 2/2 Running 0 11d alertmanager-occne-kube-prom-stack-kube-alertmanager-1 2/2 Running 65 90d occne-kube-prom-stack-grafana-78d94b6868-kwpwz 2/2 Running 2 90d occne-kube-prom-stack-kube-operator-677767c6d8-s4j67 1/1 Running 2 11d occne-kube-prom-stack-kube-state-metrics-85967b9588-csrtj 1/1 Running 2 90d occne-kube-prom-stack-prometheus-node-exporter-47n4q 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-5f8m2 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-5rvsr 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-7hs57 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-7nv7d 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-7vffn 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-8rg7c 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-kz62p 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-l2rsz 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-p662j 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-prkhl 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-q5fnf 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-r7mjv 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-t4wcm 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-vq25w 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-vz9qn 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-xlb8c 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-xnwjr 1/1 Running 1 90d occne-kube-prom-stack-prometheus-node-exporter-z564n 1/1 Running 1 90d prometheus-occne-kube-prom-stack-kube-prometheus-0 2/2 Running 3 90d prometheus-occne-kube-prom-stack-kube-prometheus-1 2/2 Running 3 90d [cloud-user@incredibles1-bastion-1 ~]$ Please let me know if anymore info is required I will attach. Thank you for your time and help. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b6fe7e0b-17a2-4da2-96bf-f6007d43a5d9n%40googlegroups.com.
