Reading this I think there is a simple solution. Lets say you go with the aggressive 5 sec file_sd interval, but you're still worried about a false alert, why not just add a "for" clause in your alerting rules?
https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ groups: - name: example rules: - alert: something expr: <metric is bad and device SHOULD be monitored> for: 30s # <--- Wait for expr to be true for 30 seconds before dispatching the alert Would that work? On Monday, December 13, 2021 at 9:17:25 PM UTC-5 [email protected] wrote: > Thank you very much Brian. > > On Monday, December 13, 2021 at 7:04:33 PM UTC+8 Brian Candler wrote: > >> I think there's not much point continuing this discussion. Prometheus >> does not have exactly what you keep asking for, which is "to check" >> something externally before sending an alert to alertmanager. All it can >> check is what's in its timeseries database at that time. >> >> However, you can achieve what you want by applying a silence to >> alertmanager. Note that you can do this *before* an alert fires: a silence >> is just a matching rule, saying "ignore alerts which match these labels". >> Push out a silence matching {instance="X"} as soon as you know you're not >> interested in alerts from device X, and you won't receive notifications for >> that device. >> >> If that doesn't meet your needs, then I'm sorry, but perhaps you should >> look for different software which does, or write something yourself. You >> could for example write a webhook receiver for alerts, which performs any >> check you like before forwarding the alert to its final destination. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1602bdd2-2676-41b3-8ded-f91898a30a4dn%40googlegroups.com.
