This move is now done. Please contact me immediately if you're having
trouble with any of your instances -- there are currently backups of
most instance files remaining on the old hardware, but I'll be cleaning
that up later in the week.
-Andrew
On 4/28/15 5:10 PM, Andrew Bogott wrote:
Executive summary:
I'm going to reboot a ton of instances at random times next week. If
you don't want me to reboot yours, email me.
Don't worry about Tools or Deployment-prep; they're already on the
'handle with care' list.
Explanation:
I've been migrating instances to new hardware like crazy, and this
morning discovered that, upon arrival on a new server, instances are
taking up MUCH more disk space than they were. In some cases, 10 or
15 times as much.
This turns out to be an issue with live migration and copy-on-write
instances. The live migration code doesn't know about
never-used-and-not-allocated-space in an instance, so when I migrate
an xlarge instance that only used 8G of disk space (but had an
allocated 160G of space), live-migrate copies that extra 152G of
emptiness, thus foiling all of our attempts to safely overprovision.
Cold migration (which involves shutting down an instance and copying
the whole VM in one lump) does not have this problem. So, two things
are going to happen:
- Instances that have not yet moved to the new hardware will be cold
migrated rather than live migrated. That means a shutdown, a
few-minute delay, and a restart.
- Large and XLarge instances that have already migrated need to be
re-shrunk to their proper copy-on-write size. That's pretty quick,
but also requires a stop and start.
If the idea of a few minutes of downtime for your instance doesn't
worry you, then you can do nothing. If you need your downtime
scheduled /or/ if downtime is unacceptable, just let me know. I can
live-migrate a few extra-precious instances and avoid downtime if
needed. Don't hesitate to ask.
None of this will start until Monday the the 4th. That gives us a
good long while to get Tools squared away, and also gives you a good
long time to notice this email :)
Sorry this upgrade has involved so many additional complications!
-Andrew
_______________________________________________
Labs-announce mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-announce
_______________________________________________
Labs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-l
