Stuart - Since the DC1 metrics are now in DC2-4's Prometheus DB and each DC has the same exporters and alert rules, DC2-4's Prometheus instance analyzed DC1 metrics (which should not be there), found the space condition in DC1 and fired an alert on DC1 metrics with the DC2-4's job name, essentially alerting 3 extra times with different a job name.
DC1 metrics should not be in DCs2-4. On Monday, June 8, 2020 at 11:31:17 AM UTC-5, Kevin Kruppa wrote: > > I recently set up a single Alertmanager instance for multiple Prometheus > instances in different datacenters. > > One datacenter has both an Alertmanager instance and a Prometheus instance > (DC1) The other datacenters are running only a Prometheus instance (DC2, > DC3, DC4) > > Each datacenter runs the same exporters under different job names and has > the same set of alert rules. Labels are different for the metrics in each > of the DCs. > > A space metric hit the alert criteria in DC1 and an alert was sent out. > Prometheus in DCs2-4 also detected DC1's space problem and fired an alert > with the metric labels from DC1 but with DC2-4's job name. > > Taking it a step further, the alert condition in DC1 was fixed but the > alert is still firing in DCs2-4. > > Looking at this is more detail, I found the metrics for DC1 are now in > DCs2-4. DC1 does not have DC2-4's metrics. > > This all started when I pointed DC2-4 to DC1 Alertmanager. > > Why would the Prometheus metrics from the DC running Alertmanager show up > in the other DCs? > > How are the metrics from DC1 getting to the other DC's database? The > other DCs do not scrape targets in DC1. > > Prometheus is running version 2.17.0-rc.4, builddate 20200321-19:08:16 > > Alertmanager is running version 0.20.0, builddate 20191211-14:13:14 > > Alertmanager config section in prometheus.yml: > > # Alertmanager configuration > > alerting: > > alertmanagers: > > - scheme: http > > path_prefix: /alertmanager1/ > > static_configs: > > - targets: [<alertmanger server>:9093'] > > > Thanks, > > Kevin > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/de811a07-7b86-464e-8a5d-db0460949388o%40googlegroups.com.
