When using file discovery: as soon as you change the contents, prometheus will pick it up and apply the new settings immediately - there's no need to signal prometheus that it has changed. Or there are other service discovery mechanisms you can use, that your inventory could update.
Now, the fact that a target has gone down doesn't necessarily mean you want to stop monitoring it entirely. If you do, you will lose a lot of information (did it come back? when did it come back? has it been down and up intermittently?). So I think you're right to be thinking about suppressing alarms, not suppressing monitoring. You've already had some notifications when the thing went down first. (Personally I disable sending of automatic repeat notifications, and automatic resolved notifications). Maybe your management system is really saying "look, we know there's a problem with this system; stop sending any more alerts about it until the problem has been investigated and solved". If that's what you're trying to do, then there's another option you can look at, which is to dynamically create "silences" in alertmanager. That is: when the inventory system has marked a machine as being in trouble, it creates a silence matching the instance name, with a defined end time (say 30 minutes in the future), and periodically keeps updating the silence for as long as the machine is out of service. When monitoring="yes" is set back, then it can delete the silence, or just let it expire. These silences can be created via the alertmanager API. There are other applications like karma <https://github.com/prymitive/karma> which talk to this API, which might give you some clues how it works. Regards, Brian. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/79dab02a-0e55-47fc-90e2-2f1c0cb342a4n%40googlegroups.com.
