Saj, this is really really awesome. On Tue, Jan 15, 2013 at 5:14 PM, Saj Goonatilleke <[email protected]> wrote:
> (It looks like my post to the list went AWOL, so here it is again.) > > On Mon Jan 14, 2013 at 19:41:48 -0500, Adam Moskowitz wrote: > > Is anyone using continuous build or continuous integration tools -- > > stuff like Jenkins or Bamboo or CruiseControl [1] -- for sysadmin tasks? > > If so, can you say what kinds of tasks you're doing with these tools? > > I'm one of a band of merry sysadmins who use BuildBot to provision > servers. > > Over time, we started provisioning so many of the darned things by hand > that, one day, we decided to invest in (almost) completely robotising > our provisioning process. We have had Kickstart (and CFEngine, later > Puppet) from Day One, but Kickstart couldn't power our machines on by > itself. Neither could it click through the monotonous forms on our > asset management and billing systems, or help us satisfy our > documentation requirements. All up, dull auxiliary tasks were consuming > hours of operator time a pop! > > Fast forward. Our entire provisioning process was automated end-to-end. > New problem: it only takes a single misbehaving mirror or a subtle > problem in a change to a firewall configuration to trip up a robot [1]. > When performing or supervising this work by hand, people don't tend to > dwell on little hiccups: they get fixed and we move on. For the first > time, we understood the significant risk our volatile environment was > placing on our automation goals. > > The problem was largely solved by replicating a subset of our live > infrastructure (DNS nameservers and resolvers; DHCP+TFTP servers; proxy > servers; Puppetmasters; backup servers; availability, performance and > intrusion monitoring systems; etc.) to a mostly isolated test > environment. BuildBot pushes make-believe server orders with varying > desired specifications through this test environment 24 hours a day > looking for failure points that we are not already monitoring. > Minute-by-minute changes from our configuration management system get > applied here, too, so we can proactively address changes that have > introduced subtle unwanted side effects on the provisioning process. > > Don't just chuck your code into the CI wringer; chuck in your whole > distributed operating environment, too. :) > > 1: http://what-if.xkcd.com/5/ > _______________________________________________ > Discuss mailing list > [email protected] > https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss > This list provided by the League of Professional System Administrators > http://lopsa.org/ > -- Joseph A Kern [email protected]
_______________________________________________ Discuss mailing list [email protected] https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
