"Set up a local development environment that mirrors the servers using virtual box, vagrant, docker, etc, there are a lot of options in the space. The main goal is to be capable of developing manifests locally that you can confidently push out to the fleet."
I gotta say this is really the golden piece of advice I've seen thus far. If you can replicate and virtualize your prod enviro to a mngmt & monitoring tool R&D enviro, that's the stuff Sys Admin dreams are made of! =) I hope you've some time & energy to share some of your learnings, experiences, discoveries, etc along the way! Heck, it might make for a great PLUG Meeting Talk someday too!
