Hi all, Update - most of the home directories have been restored at this point - we're down to the last few, that seem to have had a lot of data hence taking quite some time.
These are the pending instances(currently being restored): data.eqiad.wmflabs ldfclient.eqiad.wmflabs mathosphere.eqiad.wmflabs mwoffliner4.eqiad.wmflabs nomad.eqiad.wmflabs reading.eqiad.wmflabs recommendations.eqiad.wmflabs captcha-apiproxy-03.eqiad.wmflabs dashboardchat.eqiad.wmflabs maps-cruncher.eqiad.wmflabs pdf.eqiad.wmflabs mwoffliner[2-3].eqiad.wmflabs phlogiston-2.eqiad.wmflabs pole.eqiad.wmflabs Other than this, it should all look good. Do let us know if something seems amiss here or on #wikimedia-labs. Thanks for being patient! On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan < [email protected]> wrote: > Update: NFS for labs projects is back to read write now. > > There was an incident in the process of the migration that caused /home in > a significant number of labs instances (where /home was not on NFS) to be > replaced by an erroneous symlink. We are currently working on restoring all > the home directories - it looks like it may take at-least a couple hours. > I'll keep the list updated on progress. > > > On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan < > [email protected]> wrote: > >> Update: this is still in progress. There's some ongoing issues with home >> directories on non-nfs /home across Labs (not tools or maps) - we are >> working on it, and will update soon. Do feel free to reach out >> #wikimedia-labs if you have any questions or concerns. >> >> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan < >> [email protected]> wrote: >> >>> Reminder: This is starting soon, in ~22 minutes. >>> >>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan < >>> [email protected]> wrote: >>> >>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC). >>>> >>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < >>>> [email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> Continuing the storage redundancy and reliability efforts for Labs ( >>>>> https://phabricator.wikimedia.org/T126083), the final migration to >>>>> the new NFS storage cluster for Labs projects with NFS enabled is >>>>> upcoming. >>>>> The migration is planned to happen 2017-01-18 starting 09:00 >>>>> PST(16:00 UTC). This *does not* affect tools, maps or any other projects >>>>> that don't have /home or /data/project mounted. The migration window is >>>>> expected to be fairly short (<3 hours) - but could last up to 6 hours. >>>>> >>>>> During the migration, no new data will be written to NFS (/home or >>>>> /data/project), but existing data will be accessible in Read-only mode for >>>>> the most part. Post migration, any services or jobs that were running on >>>>> top of NFS (/home or /data/project) will require manual restarts. Jobs >>>>> running on top of /scratch or /public/dumps will be unaffected. I will >>>>> keep >>>>> the lists and #wikimedia-labs updated on progress during and after the >>>>> migration. >>>>> >>>>> The list of labs projects that will be affected in this migration are: >>>>> >>>>> >>>>> - catgraph >>>>> >>>>> >>>>> - account-creation-assistance >>>>> >>>>> >>>>> - contributors >>>>> >>>>> >>>>> - wikidata-topicmaps >>>>> >>>>> >>>>> - sugarcrm >>>>> >>>>> >>>>> - wikidumpparse >>>>> >>>>> >>>>> - video >>>>> >>>>> >>>>> - openstack >>>>> >>>>> >>>>> - testlabs >>>>> >>>>> >>>>> - wikidata-dev >>>>> >>>>> >>>>> - quarry >>>>> >>>>> >>>>> - huggle >>>>> >>>>> >>>>> - editor-engagement >>>>> >>>>> >>>>> - utrs >>>>> >>>>> >>>>> - wmt >>>>> >>>>> >>>>> - cvn >>>>> >>>>> >>>>> - fastcci >>>>> >>>>> >>>>> - toolsbeta >>>>> >>>>> >>>>> - project-proxy >>>>> >>>>> >>>>> - dumps >>>>> >>>>> >>>>> - bots >>>>> >>>>> >>>>> - snuggle >>>>> >>>>> >>>>> - math >>>>> >>>>> >>>>> - wikisource-tools >>>>> >>>>> >>>>> The tracking task on phabricator is here - >>>>> https://phabricator.wikimedia.org/T154336. Let us know if you have >>>>> any questions or concerns on the list or on #wikimedia-labs. >>>>> >>>>> -- >>>>> Madhu Viswanathan >>>>> Operations Engineer, Wikimedia Labs >>>>> >>>> >>>> >>>> >>>> -- >>>> --Madhu :) >>>> >>> >>> >>> >>> -- >>> --Madhu :) >>> >> >> >> >> -- >> --Madhu :) >> > > > > -- > --Madhu :) > -- --Madhu :)
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
