Good news:
I've now finished rebooting all the labvirt hosts, and the last few VMs
should be in the process of spinning back up right now.
Bad news:
- We have a lot more (non-VM-hosting) servers to restart! Expect brief
outages in storage, horizon, wikitech, etc during the next few hours.
- Most VMs are still running insecure kernels. I didn't upgrade them
yet because it's not entirely clear that the current generation of
security patches won't break random Java apps here and there. Once
there's a clear resolution to that issue there will be another round of
reboots, which I will announce in advance.
- The restart process hit k8s nodes a bit harder than planned, so if
your tools are running on Kubernetes you might want to verify that
things are up and running properly. If they aren't, a simple restart
should resolve things.
On 6/21/17 8:56 AM, Andrew Bogott wrote:
Reminder: These reboots will start in about 5 minutes.
On 6/20/17 11:38 AM, Andrew Bogott wrote:
Good morning!
In order to plug a newly-revealed security hole, we need to upgrade
kernels everywhere and reboot everything. I'll be doing this for all
instances and VM hosts tomorrow, 2017-06-21, beginning at 14:00UTC
(aka 7AM in San Francisco).
Project admins should expect instances to be shut down for 10-15
minutes and then restarted at some point tomorrow. I don't expect
the process to take absolutely all day, but it might. If you have an
instance that is a special case and you need it to be restarted
during a more specific window, please respond to me directly with
details about what you need and we'll try to make it happen.
These reboots should have minimal impact on things running in Tools,
although it's always good to keep an eye out -- some tools are unable
to recover from a shift between nodes and need a manual restart.
Sorry for the inconvenience!
-Andrew
_______________________________________________
Labs-announce mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-announce
_______________________________________________
Labs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-l
