Thank you Chase. :-) Everything on my instance is now restored and working, thank you local backups.
Cyberpower678 English Wikipedia Account Creation Team English Wikipedia Administrator Global User Renamer > On Jan 23, 2017, at 12:45, Chase Pettet <[email protected]> wrote: > > For onlookers, we caught up on IRC and things are good now. > > Thank you for the honest feedback Maximillian. This was an unfortunate and > unplanned outcome that we struggled to recover from. A handful of users we > were not able to restore /home for completely and it's been an uphill battle. > >> On Mon, Jan 23, 2017 at 10:42 AM, Maximilian Doerr >> <[email protected]> wrote: >> The migration went horribly wrong on my project, which deleted roughly 60% >> of the project, including the .my.cnf file. All the permissions and file >> ownerships were screwed up on the remaining files, and completely broke the >> bot tasks running on it. I have restored the permissions, ownership and >> files, but I’m still missing my db cnf file to access the DB with. It was >> not fun having to fix this. >> >> Cyberpower678 >> English Wikipedia Account Creation Team >> English Wikipedia Administrator >> Global User Renamer >> >>> On Jan 19, 2017, at 13:03, Madhumitha Viswanathan >>> <[email protected]> wrote: >>> >>> Update: 2 instances are still being restored - taxonbot.dwl and >>> pole.wikidata-query, the rest on the previous list are done. >>> >>> On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan >>> <[email protected]> wrote: >>>> Hi all, >>>> >>>> Update - most of the home directories have been restored at this point - >>>> we're down to the last few, that seem to have had a lot of data hence >>>> taking quite some time. >>>> >>>> These are the pending instances(currently being restored): >>>> >>>> data.eqiad.wmflabs >>>> ldfclient.eqiad.wmflabs >>>> mathosphere.eqiad.wmflabs >>>> mwoffliner4.eqiad.wmflabs >>>> nomad.eqiad.wmflabs >>>> reading.eqiad.wmflabs >>>> recommendations.eqiad.wmflabs >>>> captcha-apiproxy-03.eqiad.wmflabs >>>> dashboardchat.eqiad.wmflabs >>>> maps-cruncher.eqiad.wmflabs >>>> pdf.eqiad.wmflabs >>>> mwoffliner[2-3].eqiad.wmflabs >>>> phlogiston-2.eqiad.wmflabs >>>> pole.eqiad.wmflabs >>>> >>>> Other than this, it should all look good. Do let us know if something >>>> seems amiss here or on #wikimedia-labs. Thanks for being patient! >>>> >>>> >>>>> On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan >>>>> <[email protected]> wrote: >>>>> Update: NFS for labs projects is back to read write now. >>>>> >>>>> There was an incident in the process of the migration that caused /home >>>>> in a significant number of labs instances (where /home was not on NFS) to >>>>> be replaced by an erroneous symlink. We are currently working on >>>>> restoring all the home directories - it looks like it may take at-least a >>>>> couple hours. I'll keep the list updated on progress. >>>>> >>>>> >>>>>> On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan >>>>>> <[email protected]> wrote: >>>>>> Update: this is still in progress. There's some ongoing issues with home >>>>>> directories on non-nfs /home across Labs (not tools or maps) - we are >>>>>> working on it, and will update soon. Do feel free to reach out >>>>>> #wikimedia-labs if you have any questions or concerns. >>>>>> >>>>>>> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan >>>>>>> <[email protected]> wrote: >>>>>>> Reminder: This is starting soon, in ~22 minutes. >>>>>>> >>>>>>>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan >>>>>>>> <[email protected]> wrote: >>>>>>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC). >>>>>>>> >>>>>>>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan >>>>>>>>> <[email protected]> wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> Continuing the storage redundancy and reliability efforts for Labs >>>>>>>>> (https://phabricator.wikimedia.org/T126083), the final migration to >>>>>>>>> the new NFS storage cluster for Labs projects with NFS enabled is >>>>>>>>> upcoming. The migration is planned to happen 2017-01-18 starting >>>>>>>>> 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other >>>>>>>>> projects that don't have /home or /data/project mounted. The >>>>>>>>> migration window is expected to be fairly short (<3 hours) - but >>>>>>>>> could last up to 6 hours. >>>>>>>>> >>>>>>>>> During the migration, no new data will be written to NFS (/home or >>>>>>>>> /data/project), but existing data will be accessible in Read-only >>>>>>>>> mode for the most part. Post migration, any services or jobs that >>>>>>>>> were running on top of NFS (/home or /data/project) will require >>>>>>>>> manual restarts. Jobs running on top of /scratch or /public/dumps >>>>>>>>> will be unaffected. I will keep the lists and #wikimedia-labs updated >>>>>>>>> on progress during and after the migration. >>>>>>>>> >>>>>>>>> The list of labs projects that will be affected in this migration are: >>>>>>>>> >>>>>>>>> catgraph >>>>>>>>> account-creation-assistance >>>>>>>>> contributors >>>>>>>>> wikidata-topicmaps >>>>>>>>> sugarcrm >>>>>>>>> wikidumpparse >>>>>>>>> video >>>>>>>>> openstack >>>>>>>>> testlabs >>>>>>>>> wikidata-dev >>>>>>>>> quarry >>>>>>>>> huggle >>>>>>>>> editor-engagement >>>>>>>>> utrs >>>>>>>>> wmt >>>>>>>>> cvn >>>>>>>>> fastcci >>>>>>>>> toolsbeta >>>>>>>>> project-proxy >>>>>>>>> dumps >>>>>>>>> bots >>>>>>>>> snuggle >>>>>>>>> math >>>>>>>>> wikisource-tools >>>>>>>>> >>>>>>>>> The tracking task on phabricator is here - >>>>>>>>> https://phabricator.wikimedia.org/T154336. Let us know if you have >>>>>>>>> any questions or concerns on the list or on #wikimedia-labs. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Madhu Viswanathan >>>>>>>>> Operations Engineer, Wikimedia Labs >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> --Madhu :) >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> --Madhu :) >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> --Madhu :) >>>>> >>>>> >>>>> >>>>> -- >>>>> --Madhu :) >>>> >>>> >>>> >>>> -- >>>> --Madhu :) >>> >>> >>> >>> -- >>> --Madhu :) >>> _______________________________________________ >>> Labs-l mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/labs-l >> >> >> _______________________________________________ >> Labs-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/labs-l >> > > > > -- > Chase Pettet > Engineering Manager -- Labs > chasemp on phabricator and IRC > _______________________________________________ > Labs-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/labs-l
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
