[prometheus-users] Re: what to do about flapping alerts?

2024-04-08 Thread Christoph Anton Mitterer
On Monday, April 8, 2024 at 11:05:41 PM UTC+2 Brian Candler wrote: On Monday 8 April 2024 at 20:57:34 UTC+1 Christoph Anton Mitterer wrote: But for Prometheus, with keep_firing_for, it will be like the same alert. If the alerts have the exact same set of labels (e.g. the alert is at the leve

[prometheus-users] Re: what to do about flapping alerts?

2024-04-08 Thread 'Brian Candler' via Prometheus Users
On Monday 8 April 2024 at 20:57:34 UTC+1 Christoph Anton Mitterer wrote: Assume the following (arguably a bit made up) example: One has a metric that counts the number of failed drives in a RAID. One drive fails so some alert starts firing. Eventually the computing centre replaces the drive and

[prometheus-users] Re: what to do about flapping alerts?

2024-04-08 Thread Christoph Anton Mitterer
Hey Brian. On Saturday, April 6, 2024 at 9:33:27 AM UTC+2 Brian Candler wrote: > but AFAIU that would simply affect all alerts, i.e. it wouldn't just keep firing, when the scraping failed, but also when it actually goes back to an ok state, right? It affects all alerts individually, and I beli