Hello, We use prometheus , alertmanager and blackbox-exporter to check hosts if they respond to icmp. Host counts are 1K+. We noticed sometimes and randomly the alerts are not generated (prometheus dashboard --> alerts) when the hosts/targets are actually down. Restarting prometheus, alertmanager and blackbox-exports fixes the issue. Don't see anything that standouts in the logs. How do I troubleshoot and is there anything like cache data in prometheus that needs to be cleared?
Thanks Paras. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6bfb92dc-2a18-44d9-8fda-d6f84efba0e7n%40googlegroups.com.
