Update: 2 instances are still being restored - taxonbot.dwl and pole.wikidata-query, the rest on the previous list are done.
On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan < [email protected]> wrote: > Hi all, > > Update - most of the home directories have been restored at this point - > we're down to the last few, that seem to have had a lot of data hence > taking quite some time. > > These are the pending instances(currently being restored): > > data.eqiad.wmflabs > ldfclient.eqiad.wmflabs > mathosphere.eqiad.wmflabs > mwoffliner4.eqiad.wmflabs > nomad.eqiad.wmflabs > reading.eqiad.wmflabs > recommendations.eqiad.wmflabs > captcha-apiproxy-03.eqiad.wmflabs > dashboardchat.eqiad.wmflabs > maps-cruncher.eqiad.wmflabs > pdf.eqiad.wmflabs > mwoffliner[2-3].eqiad.wmflabs > phlogiston-2.eqiad.wmflabs > pole.eqiad.wmflabs > > Other than this, it should all look good. Do let us know if something > seems amiss here or on #wikimedia-labs. Thanks for being patient! > > > On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan < > [email protected]> wrote: > >> Update: NFS for labs projects is back to read write now. >> >> There was an incident in the process of the migration that caused /home >> in a significant number of labs instances (where /home was not on NFS) to >> be replaced by an erroneous symlink. We are currently working on restoring >> all the home directories - it looks like it may take at-least a couple >> hours. I'll keep the list updated on progress. >> >> >> On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan < >> [email protected]> wrote: >> >>> Update: this is still in progress. There's some ongoing issues with home >>> directories on non-nfs /home across Labs (not tools or maps) - we are >>> working on it, and will update soon. Do feel free to reach out >>> #wikimedia-labs if you have any questions or concerns. >>> >>> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan < >>> [email protected]> wrote: >>> >>>> Reminder: This is starting soon, in ~22 minutes. >>>> >>>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan < >>>> [email protected]> wrote: >>>> >>>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC). >>>>> >>>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < >>>>> [email protected]> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Continuing the storage redundancy and reliability efforts for Labs ( >>>>>> https://phabricator.wikimedia.org/T126083), the final migration to >>>>>> the new NFS storage cluster for Labs projects with NFS enabled is >>>>>> upcoming. >>>>>> The migration is planned to happen 2017-01-18 starting 09:00 >>>>>> PST(16:00 UTC). This *does not* affect tools, maps or any other projects >>>>>> that don't have /home or /data/project mounted. The migration window is >>>>>> expected to be fairly short (<3 hours) - but could last up to 6 hours. >>>>>> >>>>>> During the migration, no new data will be written to NFS (/home or >>>>>> /data/project), but existing data will be accessible in Read-only mode >>>>>> for >>>>>> the most part. Post migration, any services or jobs that were running on >>>>>> top of NFS (/home or /data/project) will require manual restarts. Jobs >>>>>> running on top of /scratch or /public/dumps will be unaffected. I will >>>>>> keep >>>>>> the lists and #wikimedia-labs updated on progress during and after the >>>>>> migration. >>>>>> >>>>>> The list of labs projects that will be affected in this migration are: >>>>>> >>>>>> >>>>>> - catgraph >>>>>> >>>>>> >>>>>> - account-creation-assistance >>>>>> >>>>>> >>>>>> - contributors >>>>>> >>>>>> >>>>>> - wikidata-topicmaps >>>>>> >>>>>> >>>>>> - sugarcrm >>>>>> >>>>>> >>>>>> - wikidumpparse >>>>>> >>>>>> >>>>>> - video >>>>>> >>>>>> >>>>>> - openstack >>>>>> >>>>>> >>>>>> - testlabs >>>>>> >>>>>> >>>>>> - wikidata-dev >>>>>> >>>>>> >>>>>> - quarry >>>>>> >>>>>> >>>>>> - huggle >>>>>> >>>>>> >>>>>> - editor-engagement >>>>>> >>>>>> >>>>>> - utrs >>>>>> >>>>>> >>>>>> - wmt >>>>>> >>>>>> >>>>>> - cvn >>>>>> >>>>>> >>>>>> - fastcci >>>>>> >>>>>> >>>>>> - toolsbeta >>>>>> >>>>>> >>>>>> - project-proxy >>>>>> >>>>>> >>>>>> - dumps >>>>>> >>>>>> >>>>>> - bots >>>>>> >>>>>> >>>>>> - snuggle >>>>>> >>>>>> >>>>>> - math >>>>>> >>>>>> >>>>>> - wikisource-tools >>>>>> >>>>>> >>>>>> The tracking task on phabricator is here - >>>>>> https://phabricator.wikimedia.org/T154336. Let us know if you have >>>>>> any questions or concerns on the list or on #wikimedia-labs. >>>>>> >>>>>> -- >>>>>> Madhu Viswanathan >>>>>> Operations Engineer, Wikimedia Labs >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --Madhu :) >>>>> >>>> >>>> >>>> >>>> -- >>>> --Madhu :) >>>> >>> >>> >>> >>> -- >>> --Madhu :) >>> >> >> >> >> -- >> --Madhu :) >> > > > > -- > --Madhu :) > -- --Madhu :)
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
