On Wed, 25 Nov, 2020, 8:26 pm Stuart Clark, <[email protected]> wrote:
> How many Alertmanager instances are there? Can they talk to each other and > is Prometheus configured and able to push alerts to them all? > >> Single instance as of now. I did setup a Alertmanager Mesh of 2 Alertmanagers but I am facing duplicate alert issue in that setup. Another issue that is pending for me. Hence, currently only a single Alertmanager is receiving alerts from my Prometheus instance. On 25 November 2020 14:07:41 GMT, "Yagyansh S. Kumar" < > [email protected]> wrote: >> >> Hi Stuart. >> >> On Wed, 25 Nov, 2020, 6:56 pm Stuart Clark, <[email protected]> >> wrote: >> >>> On 25/11/2020 11:46, [email protected] wrote: >>> > The alert formation doesn't seem to be a problem here, because it >>> > happens for different alerts randomly. Below is the alert for Exporter >>> > being down for which it has happened thrice today. >>> > >>> > - alert: ExporterDown >>> > expr: up == 0 >>> > for: 10m >>> > labels: >>> > severity: "CRITICAL" >>> > annotations: >>> > summary: "Exporter down on *{{ $labels.instance }}*" >>> > description: "Not able to fetch application metrics from *{{ >>> > $labels.instance }}*" >>> > >>> > - the ALERTS metric shows what is pending or firing over time >>> > >> But the problem is that one of my ExporterDown alerts is active >>> > since the past 10 days, there is no genuine reason for the alert to go >>> > to a resolved state. >>> > >>> What do you have evaluation_interval set to in Prometheus, and >>> resolve_timeout in Alertmanager? >>> >> >> My evaluation interval is 1m whereas my scrape timeout and scrape >> interval are 25s. Resolve timeout in Alertmanager is 5m. >> >>> >>> Is the alert definitely being resolved, as in you are getting a resolved >>> email/notification, or could it just be an email/notification for a long >>> running alert? - you should get another email/notification every now and >>> then based on repeat_interval. >>> >> >> Yes, I suspected that too in the beginning but I am logging each and >> every alert notification and found that I am indeed getting resolved >> notification for that alert and again firing notification the very next >> second. >> >>> >>> >>> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAFGi5vDWArLwvvk_Uyg-5TAMVMrLUkjM%3Dqm%3DZZJnnG1hF6mNQw%40mail.gmail.com.
