tl;dr: scratch volumes were broken for about 45 minutes. Things are now
resolved or resolving, and no action is needed on your part.
--
The transfer to the new scratch server did not go as smoothly as I
hoped. Any toolforge jobs or VM processes that accessed the scratch
volume were probably hanging or crashing for about 45 minutes, from
around 15:15 UTC to 16:00 UTC.
This has now been resolved on all cloud-vps hosts, but some toolforge
jobs are probably still stuck. We are gradually rescheduling all pods,
so no action is needed on your part other than a bit more waiting.
More details can be found on https://phabricator.wikimedia.org/T433426 ;
eventually a full incident report will be published and linked to from
that task.
-Andrew
On 7/28/26 3:17 PM, Andrew Bogott wrote:
This email relates to the shared /data/scratch mount used by toolforge
and some cloud-vps projects. It is provided to the following projects:
- dumps
- language
- maps
- math
- mwoffliner
- paws
- statanalyser
- testlabs
- tools
- toolsbeta
- video
- wikidata-query
- wikidumpparse
- wikiqlever
As part of our push to eliminate the use of Debian Bullseye, we will
be replacing the server that hosts the nfs scratch volume on Monday,
sometime around 15:00 UTC. Testing suggests that this will only
produce a few minutes of downtime for the volume (likely reported as
'Permission Denied'); if you find your system frozen please give it a
few minutes and then reboot if waiting doesn't help.
The data on the old server will be transferred to the new server, so
no data will be lost.[0]
Bonus points: If you are a cloud-vps admin and have read this far,
please consider whether you still need the scratch volume. This
service was initially provided before we had attachable block storage
available[1], so many users rely on scratch not for file transfer
between hosts but because they needed extra storage space. If you are
one of those cases, please open a quota ticket[2] and mention that you
are moving off of scratch. Life without NFS is better for everyone!
This work is tracked on phabricator here:
https://phabricator.wikimedia.org/T433426
- Andrew + the WMCS team
[0] ...but if you have data on scratch that you care about, it should
be somewhere else! We make no reliability guarantees for this storage.
[1]
https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances
[2] https://phabricator.wikimedia.org/project/view/2880/
_______________________________________________
Cloud-announce mailing list -- [email protected]
List information:
https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/
