Yesterday, the filesystem used by many Labs tools suffered a catastrophic failure, causing most tools to break. This was noticed quickly but recovery is taking a long time because of the size of the filesystem.
There have been file system corruption on the filesystem backing the NFS setup that all of labs uses, causing a prolonged outage. The Operations team is currently attempting to restore a backup made on June 9 at 16:00 UTC. Recovery of modifications made after that date is potentially possible, but our first priority is getting the backup restored. We will update the incident report page https://wikitech.wikimedia.org/wiki/Incident_documentation/20150617-LabsNFSOutage with notes on our progress. E-mails will also be sent to the labs-announce (https://lists.wikimedia.org/mailman/listinfo/labs-announce) and labs-l (https://lists.wikimedia.org/mailman/listinfo/labs-l) on significant changes. We are not yet able to estimate when things will be back up fully. This also means that tools hosted on tools.wmflabs.org will not be accessible until this is finished, and even then they might need some more fiddling to work properly. We will update https://wikitech.wikimedia.org/wiki/Incident_documentation/20150617-LabsNFSOutage as well as soon as we have more information. If you have a non-tools project on labs that does not depend on NFS and is currently down, you can recover it by getting rid of NFS. (We can help you with that.) For instructions, see https://wikitech.wikimedia.org/wiki/Recover_instance_from_NFS . Join us on #wikimedia-labs and we will assist you. -- Yuvi Panda T http://yuvi.in/blog _______________________________________________ Labs-announce mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-announce _______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
