On Wed, Jan 13, 2010 at 4:45 PM, Bryce T. Pier <[email protected]> wrote: > For years my employer has only patched *nix systems on an annual basis. > We've now been directed to apply security patches quarterly. Due to the > infrequency of patching in the past, there has developed a fairly high > level of paranoia around patching "breaking" things, particularly > related to servers not coming back from the post-patch reboot. To > mitigate these fears I've been asked to document procedures for > backing-out the applied patches and/or recovering the server in the > event of one not coming back up.
This is why regular patching, regular reboots, regular reinstalls, should be the norm. Uptime contests are definitely in the past. > So I'm curious what other people are doing on the Linux platform. > > Do you use root disk mirrors and break the mirror prior to patching? > Do you utilize filesystem dumps (dumpe2fs, etc) or rely on enterprise > backups of the OS filesystems? > Do you use rpm rollbacks? > Rebuild / re-image the server if there are problems? - Separate your data, make sure they are backed up. - Make sure you are able to reinstall a server from scratch, pull the necessary packages and configuration easily, then restore data backups. If you can afford test hardware, it makes things easier. - If you can afford extra hardware, you can rotate your roles while taking one machine at a time for upgrade - With extra hardware, testing or canary is just a matter of pulling the extra machine and installing the updated software, kernel or distribution - With extra hardware you don't have to fear so much a hardware failure. The extra machine takes the place of the failed one while you repair. - I am starting to run out of things you can do with extra hardware :) Of course this is much easier with web frontends or LDAP slaves than it is with a single, monolithic database server, but even with a DB you should be able to easily setup a second machine, pull your backups and run it. Once you are at this point, you do not fear patching so much because you can prepare for it *all the time*, and you get extra benefits too. Once you are ready for that, the next logical step is to run on virtual hardware. -- olive _______________________________________________ Discuss mailing list [email protected] http://lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
