I'm thinking about ways we can reduce noisy alerts. One of the problems is it's tricky to tweak alert thresholds without any data on the precision and recall of the alert. It's a non-trivial problem to get this data because a human is typically required to classify an alert as a true positive or a false negative. This makes it hard to fully automate gathering this data. I am considering whether there is a way of obtaining this data using a hybrid approach: a human is able to classify an alert as a true positive of false positive - for example via a button in the alert body (e.g. in Slack or PagerDuty) and this gets sent to an analytics database which we can later prioritise which alert thresholds that need tweaking.
My question is, is there any precedent for this kind of system in the Prometheus/Alertmanager ecosystem? i.e. open source software that does this out of the box, or experience report blog posts? Many thanks, Will -- -- *This email is confidential and protected by copyright, and might contain privileged information. The same goes for any attachments.* * * *If we’ve sent it to you by mistake (sorry), please don’t copy it or show it to anyone. You also shouldn’t use it to make a decision, and you shouldn’t rely on the contents. Let the sender know as soon as you can, and then delete the email. Thank you!* Monzo Bank Limited is a company registered in England and Wales (No. 09446231) registered at Broadwalk House, 5 Appold St, London, EC2A 2AG. Monzo Bank Ltd is authorised by the Prudential Regulation Authority (PRA) and regulated by the Financial Conduct Authority and the PRA. Our Financial Services Register number is 730427. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e1294440-6957-435f-ab48-de541643b687n%40googlegroups.com.
