Hi all, Tools is now back to read write, and all of tools should work as expected. Webservice logs aren't being written to disk currently, and will not be written for ~12 more hours, while we make sure everything else is working alright.
The storage infrastructure is brand new, so we recommend being a bit gentle for the next few days, as we monitor performance and make sure it's all okay. Do let us know if things aren't working like you'd expect, on #wikimedia-labs/on list or phabricator. Thanks for being patient through this long maintenance window! On Mon, Nov 14, 2016 at 10:15 AM, Madhumitha Viswanathan < [email protected]> wrote: > Most of tools should be back up now. Expected behavior for the next many > hours (Expected ~12-14 hours based on last sync of tools data, but might be > much longer) > > Quoting from previous announcement [1]: > > * The tools NFS share (/data/project and /home) will be read-only for the >> duration of the maintenance, so no new data or logs will get written to it. >> * New jobs cannot be submitted for the whole maintenance window - this >> means submitting jobs through cron or tools-mail will not function, >> although tools-mail can continue to send emails. >> * Current jobs might keep running, but won't get rescheduled if they die. >> If they do not die and aren't writing to NFS they should be fine. > > > [1]: https://lists.wikimedia.org/pipermail/labs-l/2016-October/004698.html > > On Mon, Nov 14, 2016 at 9:31 AM, Madhumitha Viswanathan < > [email protected]> wrote: > >> All of tools is down currently - unexpected side effect of the ongoing >> maintenance - we are working on it, I will update once it's back up. >> Apologies for the inconvenience caused. >> >> On Mon, Nov 14, 2016 at 7:59 AM, Madhumitha Viswanathan < >> [email protected]> wrote: >> >>> Hi all, >>> >>> This is starting in a few minutes. You can follow along progress on >>> #wikimedia-labs or at T146154 >>> <https://phabricator.wikimedia.org/T146154>. >>> >>> >>> On Fri, Nov 11, 2016 at 10:16 AM, Madhumitha Viswanathan < >>> [email protected]> wrote: >>> >>>> Hi all, >>>> >>>> This migration is scheduled to happened on Monday 11/14. We will start >>>> at 08:00 PST/16:00 UTC. More details here - >>>> https://phabricator.wikimedia.org/T146154. >>>> >>>> On Tue, Nov 1, 2016 at 11:47 AM, Madhumitha Viswanathan < >>>> [email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> We had previously announced a disruptive maintenance for tools[1] on >>>>> 11/2(tomorrow), but we are going to have to postpone it tentatively to >>>>> 11/14/2016. >>>>> >>>>> Apologies for the late notice, but the server in our secondary >>>>> datacenter in Dallas that hosts data backups is having issues since >>>>> yesterday[2], and we are currently still debugging and fixing the problems >>>>> there. We've decided not to proceed with this migration until our backups >>>>> there are online. >>>>> >>>>> As always, do let us know if there are any questions/concerns here or >>>>> on #wikimedia-labs. >>>>> >>>>> [1]: https://lists.wikimedia.org/pipermail/labs-l/2016-Octob >>>>> er/004698.html >>>>> [2]: https://phabricator.wikimedia.org/T149567 >>>>> >>>> >>>> >>>> >>>> -- >>>> --Madhu :) >>>> >>> >>> >>> >>> -- >>> --Madhu :) >>> >> >> >> >> -- >> --Madhu :) >> > > > > -- > --Madhu :) > -- --Madhu :)
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
