Thanks all. This is all really helpful. pint looks useful in general, and it also looks like it will be helpful for this particular use case. Thanks for sharing! For future readers, I think the docs here are most helpful: https://github.com/cloudflare/pint/blob/main/docs/CONFIGURATION.md#alerts.
On Thursday, 30 September 2021 at 15:17:20 UTC+1 l.mi...@gmail.com wrote: > https://github.com/cloudflare/pint can try to estimate the number of time > an alert would trigger if you configure it to do so, > see example at > https://github.com/cloudflare/pint/blob/main/docs/examples/config.hcl#L58 > > Not exactly what you're looking for, but it can be useful to find alerts > that would fire too often or would never fire (if you know it should fire). > > On Tuesday, 28 September 2021 at 09:20:31 UTC+1 wills...@monzo.com wrote: > >> I'm thinking about ways we can reduce noisy alerts. One of the problems >> is it's tricky to tweak alert thresholds without any data on the precision >> and recall of the alert. It's a non-trivial problem to get this data >> because a human is typically required to classify an alert as a true >> positive or a false negative. This makes it hard to fully automate >> gathering this data. I am considering whether there is a way of obtaining >> this data using a hybrid approach: a human is able to classify an alert as >> a true positive of false positive - for example via a button in the alert >> body (e.g. in Slack or PagerDuty) and this gets sent to an analytics >> database which we can later prioritise which alert thresholds that need >> tweaking. >> >> My question is, is there any precedent for this kind of system in >> the Prometheus/Alertmanager ecosystem? i.e. open source software that does >> this out of the box, or experience report blog posts? >> >> Many thanks, >> Will >> > -- -- *This email is confidential and protected by copyright, and might contain privileged information. The same goes for any attachments.* * * *If we’ve sent it to you by mistake (sorry), please don’t copy it or show it to anyone. You also shouldn’t use it to make a decision, and you shouldn’t rely on the contents. Let the sender know as soon as you can, and then delete the email. Thank you!* Monzo Bank Limited is a company registered in England and Wales (No. 09446231) registered at Broadwalk House, 5 Appold St, London, EC2A 2AG. Monzo Bank Ltd is authorised by the Prudential Regulation Authority (PRA) and regulated by the Financial Conduct Authority and the PRA. Our Financial Services Register number is 730427. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e9e0113d-3a0f-4662-9c4b-c2fb138c3461n%40googlegroups.com.
