Le vendredi 06 juin 2014 à 14:36 +0200, Ewoud Kohl van Wijngaarden a écrit : > On Fri, Jun 06, 2014 at 01:29:44PM +0200, Michael Scherer wrote: > > Due to CVE on openssl and on kernel, I did upgrade various piece of the > > infrastructure ( foreman, lists, stats, monitoring ), which implied a > > few reboots ( due to kernel lagging behind, which is not that great with > > local root exploit ). As this is friday and I assumed most of the Tel > > Aviv office was not working, i hope this kept the disruption to a > > minimum. However, if something is broken, please tell it so we can fix. > > Nice work.
> > This also got me thinking. In order to bring a bit more order, what > > about having a fixed schedule for upgrade ? > > > > In my previous position, we were doing that once per month ( except > > during end of quarter freeze ), with mandatory reboot ( cause if > > something do not boot, you want to know it when you have a planned > > outage, not when everyone is running around updating stuff ). Fedora has > > a rather complex procedure to decide what to upgrade, hilighted on > > http://infrastructure.fedoraproject.org/infra/docs/massupgrade.txt > > At my previous employer we had something similar. I also wrote a puppet > custom fact reboot_needed which checks if the running kernel matches the > default kernel that would be booted. We also want to restart services that are affected. Hence a reboot would be simpler from a engineering point of view. > > So we could adopt a schedule ( once per month, unless there is something > > critical, in which case we do it ASAP, with warning on the list and irc > > ). > > +1 > > > The schedule should of course take in account "business need", which is > > "release schedule of ovirt". > > > > So what about "first friday of the month, unless exception" ? > > We should also make sure that we don't reboot ALL servers at once. So if > we have multiple centos 6 jenkins slaves, try to just reboot one at a > time. Also would be nice if the slave did proper scheduling in jenkins > so no jobs are running. Yep, proper orchestration would be nice. Now, if jenkins is resilient enough ( ie, if it can survive a 5 minutes downtime ), it may not be that business critical to have it running all the time. I am not a jenkins expert, not even a user, so I defer to David for that. > > And by update, i mean "yum upgrade -y". Cleaning the list of repo on > > various servers is also IMHO another task to discuss, to make sure the > > task can be safely executed. ( having something like > > mcollective/ansible/func is also needed, but that's more a convenience > > than a requirement at this stage ). > > We sometimes have pinned versions on jenkins build slaves. That means we > should either do a proper yum versionlock or find something else. Note > that I'm all in favor of being able to to a blind yum upgrades. -- Michael Scherer Open Source and Standards, Sysadmin
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
