if some one changes the status to in maintenance and performs a reboot ... prometheus will notice this status update after 3 min and during this time it could generates fake alerts
On Monday, December 13, 2021 at 5:10:51 PM UTC+8 nina guo wrote: > Thank you Brian. > > Let's assume the refresh interval is only 5 secs, but during 5 secs the > state might be changed. For exmaple, the target is changed to > maintence(which is not the state we want to monitor) and start a reboot, > but at this time point, prometheus doesn't know the state change, a fake > alert will also be triggered for the reboot action. But the maintenance > state is not the state we are going to monitor, we don't want to receive > such kind of alerts. So we would like to check if there is a way for > Prometheus to know the target state instantly before triggering an alert. > > On Monday, December 13, 2021 at 4:50:55 PM UTC+8 Brian Candler wrote: > >> You can always read the inventory as often as you like. Read it every 5 >> seconds if you like. Write it to a new file like "targets.new", compare it >> with the old "targets" file, and if they are different rename targets.new >> to targets. Then prometheus will pick it up the new file immediately. >> >> The third solution I gave you (creating silences) might work better for >> you. You can create silences whenever you like, and they take effect >> immediately. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/95de0d51-f0a6-4f13-b444-fd1a85c61567n%40googlegroups.com.
