On Wed, Jan 13, 2010 at 10:45 AM, Bryce T. Pier <[email protected]> wrote: > For years my employer has only patched *nix systems on an annual basis. > We've now been directed to apply security patches quarterly. Due to the > infrequency of patching in the past, there has developed a fairly high > level of paranoia around patching "breaking" things, particularly > related to servers not coming back from the post-patch reboot. To > mitigate these fears I've been asked to document procedures for > backing-out the applied patches and/or recovering the server in the > event of one not coming back up. > > Given that tools like RHN Satellite or Novell Zenworks don't have the > ability to do extensive pre-patch preparations like breaking hardware > root mirrors or running filesystem dumps, I have the impression that at > least in enterprise Linuxes there aren't frequently issues caused by > normal, regular patching activities. > > So I'm curious what other people are doing on the Linux platform. > > Do you use root disk mirrors and break the mirror prior to patching? > Do you utilize filesystem dumps (dumpe2fs, etc) or rely on enterprise > backups of the OS filesystems? > Do you use rpm rollbacks? > Rebuild / re-image the server if there are problems? > > Additionally, have you experienced many instances of patching tanking an > enterprise Linux server in the last couple of years? > > Thanks much! > -- > Bryce T. Pier > [email protected] > UNIX Geek
For the concerns about reboot, reboot the server before doing the patch, so you can be sure it was the patch that broke it, and not something else. Linux kernel updates add the new kernel to the boot menu and set it as default. If you have a kernel boot issue, you can easily reboot and use the menu to select the last known working kernel. As for issues with libraries and other updates, that should be addressed in your patching process. You should be patching your less critical systems first, then seeing that they don't break, then patching the production ones. Others might have ideas about other issues like libraries and things like that. You might be able to use LVM to make snapshots. Oliver also has a good point about patching frequency. The more you patch the more you get comfortable with the process. As long as you are using an Enterprise Linux distro, any updates within the same major/minor version should not cause any issues, as that is the point of an Enterprise distro. If you're not using an Enterprise distro, forget about patching, start a project to migrate all of your servers to one, and recommend the boss fires the guy who chose Fedora or Ubuntu (non-LTS) to run servers with. The only issues that show up in Enterprise linux patching is if you're going to another minor version of the OS, like 5.3 -> 5.4. Often there are some changes made that are generally larger than the simple patches, but not hugely so. Those or something you should make sure to test before rolling out in Production. _______________________________________________ Discuss mailing list [email protected] http://lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
