Update:
I upgraded the kernel on all of the new hardware, and I'm /pretty sure/
that that fixed the problem. I'm going to wait a few days and let the
dust settle -- if everything looks good over the weekend then I'll
resume scripted migration on Monday.
Apologies for any inconvenience caused by flappy VMs.
-Andrew
On 4/23/15 4:17 PM, Andrew Bogott wrote:
We've encountered an unexpected problem with this -- there's a kernel
bug running on the new hardware which is causing instances to behave
poorly.
So, I need to upgrade the kernel and reboot. I'm moving all tools
instances out of the way first so they aren't hit by the reboot; a few
other projects (notably deployment-prep and staging) will suffer
rolling reboots.
Fortunately this issue appeared early enough that most instances are
still running on the old hardware, so most of you will be unaffected.
-Andrew
On 4/22/15 11:09 AM, Andrew Bogott wrote:
Greetings!
I'll be gradually moving most labs instances to new hardware over the
coming 7-10 days. For virtually all instances this move will be
invisible to users -- the worst case scenario is that an instance
will freeze for a minute or two during the final post-copy sync. I've
already moved several projects without incident.
Nevertheless, services which have extremely touchy timeouts may error
out or throw warnings. For example, a few things in deployment-prep
sent 'service down' alerts which lasted a few seconds before being
resolved. So if you see something like that, it's probably a result
of the move.
I will be moving the Tools instances first, starting tomorrow morning
(approximately 14:00 UTC). The tools migration will take around 18
hours. On Friday morning I'll start a scripted move of all other
instances; the complete move will take around 7 days.
If this process concerns you and you'd like your instances moved at a
pre-determined time, feel free to contact me off-list and we can make
an appointment for your project.
-Andrew
_______________________________________________
Labs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-l
