On Wed, Jan 13, 2010 at 09:45:06AM -0600, Bryce T. Pier wrote: > least in enterprise Linuxes there aren't frequently issues caused by > normal, regular patching activities. > > So I'm curious what other people are doing on the Linux platform. > > Do you use root disk mirrors and break the mirror prior to patching? > Do you utilize filesystem dumps (dumpe2fs, etc) or rely on enterprise > backups of the OS filesystems? > Do you use rpm rollbacks? > Rebuild / re-image the server if there are problems? > > Additionally, have you experienced many instances of patching tanking an > enterprise Linux server in the last couple of years?
We're primarily an RHEL site, running RHEL 3, 4 and 5... We patch quarterly unless it's a specific bug (especially security) that we believe needs to be applied sooner. Our quarterly patching does include a backout plan. Problems from the update process are quite rare. Usually they're the kind of minor problem you can resolve without a true "backout". We have sometimes had problems where a kernel update didn't work and the system wouldn't come up from the post-patch reboot, usually related to third-party drivers that don't handle kernel updates well (or at all). When it happens, the system locks up during boot failing to find the OS filesystem, then you reboot again and choose the old kernel and all is well. (then either figure out and fix the problem, or simply uninstall the newer kernel, so that a later reboot won't need manual intervention). Because RedHat keeps your old kernels by default, it's much less likely that patching will render a system unbootable, though it can make booting inconvenient (needing to log into the lights out or actually go into the data center). We generally use hardware RAID on our linux servers, and we never mess with it for the purpose of patching. We have enterprise backups of most systems and that's a last-resort backout plan that's listed as a possibility in my backout procedure, but we haven't actually had to use that procedure because of regular OS updates. We have used RPM rollbacks in the past, but they've since been deprecated. Don't use them. They may not even be available on your systems anymore. It's better to get the old version of the RPM and install that. The repackaging for RPM rollbacks is inherently unreliable, and you can always fetch the previous versions of the RPM, even if you end up logging into the RHN website with a browser to find it. /var/log/rpmpkgs (or one of the rotated versions) will show you the exact package version you had before. I haven't had to because of patching, but for many of our systems we could rebuild or re-image easily. (in particular, the systems that don't have enterprise backups are all ones that are trivial to rebuild) It's a backout plan for some systems. One other thing we do, however, is handle the quarterly patching in 3 "batches". The first batch is systems with only internal customers (especially various test and development systems) that are either easy to rebuild or where extended downtime isn't a major problem for anybody; this. The second and third batches split redundant pairs (or larger redundant sets) up where possible, so if something does break a server after 12 hours, the other server can probably handle the load while we fix the first one. There's at least 48 hours between batches. One problem with this batched approach though, is that inevitably RedHat will release a huge amount of patches in between our batches. We have a few test systems on RedHat's "fastrack" channels so that we're testing packages before they make it to production systems, which helps mitigate that last problem. -- Eric Eisenhart <*[email protected]> http://eric.eisenhart.name/ IRC: freih...@freenode, AIM/yahoo: falschfreiheit Jabber/GTalk: [email protected] _______________________________________________ Discuss mailing list [email protected] http://lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
