(It looks like my post to the list went AWOL, so here it is again.) On Mon Jan 14, 2013 at 19:41:48 -0500, Adam Moskowitz wrote: > Is anyone using continuous build or continuous integration tools -- > stuff like Jenkins or Bamboo or CruiseControl [1] -- for sysadmin tasks? > If so, can you say what kinds of tasks you're doing with these tools?
I'm one of a band of merry sysadmins who use BuildBot to provision servers. Over time, we started provisioning so many of the darned things by hand that, one day, we decided to invest in (almost) completely robotising our provisioning process. We have had Kickstart (and CFEngine, later Puppet) from Day One, but Kickstart couldn't power our machines on by itself. Neither could it click through the monotonous forms on our asset management and billing systems, or help us satisfy our documentation requirements. All up, dull auxiliary tasks were consuming hours of operator time a pop! Fast forward. Our entire provisioning process was automated end-to-end. New problem: it only takes a single misbehaving mirror or a subtle problem in a change to a firewall configuration to trip up a robot [1]. When performing or supervising this work by hand, people don't tend to dwell on little hiccups: they get fixed and we move on. For the first time, we understood the significant risk our volatile environment was placing on our automation goals. The problem was largely solved by replicating a subset of our live infrastructure (DNS nameservers and resolvers; DHCP+TFTP servers; proxy servers; Puppetmasters; backup servers; availability, performance and intrusion monitoring systems; etc.) to a mostly isolated test environment. BuildBot pushes make-believe server orders with varying desired specifications through this test environment 24 hours a day looking for failure points that we are not already monitoring. Minute-by-minute changes from our configuration management system get applied here, too, so we can proactively address changes that have introduced subtle unwanted side effects on the provisioning process. Don't just chuck your code into the CI wringer; chuck in your whole distributed operating environment, too. :) 1: http://what-if.xkcd.com/5/ _______________________________________________ Discuss mailing list [email protected] https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
