On Thu, Jun 4, 2020, at 11:09, [email protected] wrote: > Hi, colleagues! > I setup and configured Alertmanager for getting email notifications, but all > of my alerts about different environments are gathered in one email and > sorted by alphabetical order by job_name. In test environment there are > approx. 250 alerts in one email. It doesn't work for urgents reaction. > Prometheus rules have 3 rules per 1 env. Now I'm testing 7 envs.
It's not useful to have 250 alerts firing at once in testing or in a real outage. You need to find something less noisy to alert on. As an example, if you have a fleet of 100 servers, you might want to send a single alert if more than 10 of them are down; then use a dashboard to figure out which ones (and why). As others have mentioned you'll also want to group your alerts; at the very least separate them by environment. -- Harald -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/22c45cf4-7000-4405-9532-c2bfbceaf601%40www.fastmail.com.
