" Are there any good FOSS software tools to help organize and monitor a mess like this?
Yes, I've used Nagios, Cacti and Zabbix in the past. Nagios can do everything you want to do and probably tons more. It has plugins, APIs, scripts, and a huge ecosystem. There is both a FOSS and commercial product. It's in the Debian pkg repos. You could install on your pc and start learning about it. https://www.nagios.org/projects/nagios-core/ There's also Cacti, which is it simpler and based on your skill set might be a better tool. https://www.cacti.net/info/cacti https://www.zabbix.com/ My background is mostly with Network Management & Monitoring as a Network Engineer with some basic Linux System Admin. I only used the packaged features & functionalities with a few basic plugins. If I could set these tools up, make them work in undocumented, non-standardized, non-homogeneous enviros, you shouldn't have any problems. One you get SNMP running, talking and collecting data that you can access via a web interface, you'll quickly gain visibility into the system and the sky is the limit. Based on past experience, I'd recommend just starting with 1 server and see what these tools can do, what works for you & your enviro, etc. All these tools were pretty good, very useful and helpful to me some 10 years ago. I suspect they're more feature & function rich. easier to setup, use, maintain and with a larger support community.
