Am 15.11.2016, 06:39, Madhumitha Viswanathan <[email protected]> schrieb:
Hi all,
Tools is now back to read write, and all of tools should work as expected. Webservice logs aren't being written to disk currently, and will not be written for ~12 more hours, while we make sure everything else is working alright.
The storage infrastructure is brand new, so we recommend being a bit gentle for the next few days, as we monitor performance and make sure it's all okay. Do let us know if things aren't working like you'd expect, on #wikimedia-labs/on list or phabricator.
Thanks for being patient through this long maintenance window!
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
On Mon, Nov 14, 2016 at 10:15 AM, Madhumitha Viswanathan <[email protected]> wrote:
Most of tools should be back up now. Expected behavior for the next many hours (Expected ~12-14 hours based on last sync of tools data, but might be much longer)
Quoting from previous announcement [1]:
* The tools NFS share (/data/project and /home) will be read-only for the duration of the maintenance, so no new data or logs will get written to it.
* New jobs cannot be submitted for the whole maintenance window - this means submitting jobs through cron or tools-mail will not function, although tools-mail can continue to send emails.
* Current jobs might keep running, but won't get rescheduled if they die. If they do not die and aren't writing to NFS they should be fine.--
On Mon, Nov 14, 2016 at 9:31 AM, Madhumitha Viswanathan <[email protected]> wrote:
All of tools is down currently - unexpected side effect of the ongoing maintenance - we are working on it, I will update once it's back up. Apologies for the inconvenience caused.
On Mon, Nov 14, 2016 at 7:59 AM, Madhumitha Viswanathan <[email protected]> wrote:
Hi all,
This is starting in a few minutes. You can follow along progress on #wikimedia-labs or at T146154.
On Fri, Nov 11, 2016 at 10:16 AM, Madhumitha Viswanathan <[email protected]> wrote:
Hi all,
This migration is scheduled to happened on Monday 11/14. We will start at 08:00 PST/16:00 UTC. More details here - https://phabricator.wikimedia.org/T146154.--
On Tue, Nov 1, 2016 at 11:47 AM, Madhumitha Viswanathan <[email protected]> wrote:
Hi all,
We had previously announced a disruptive maintenance for tools[1] on 11/2(tomorrow), but we are going to have to postpone it tentatively to 11/14/2016.
Apologies for the late notice, but the server in our secondary datacenter in Dallas that hosts data backups is having issues since yesterday[2], and we are currently still debugging and fixing the problems there. We've decided not to proceed with this migration until our backups there are online.
As always, do let us know if there are any questions/concerns here or on #wikimedia-labs.
--Madhu :)
--
--Madhu :)
--
--Madhu :)
--Madhu :)
--
--Madhu :)
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
