hi I have some batch process that push metrics to push-gateway. the batch process runs every-day for ~2 minutes. i also have alerts on the metrics, and it seems to work fine except to the following scenario.
when alert is fired, there is nothing that "clears" the alert from Prometheus, and the alert manager keeps sending the notifications. the alert can be cleared only after 24 hours, when the batch process is triggered again. i was hoping that "resolve_timeout: 5m" will solve this, but it's not. any idea how deal with such senario? thanks -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/db43506f-0f92-49de-b146-7880efa6b0ben%40googlegroups.com.
