On 2017-07-02 some users experienced a loss of data in their projects. We estimate 126 Tools out of 1,766 saw at least one file removed. We have reason to believe a user acting under a Tool account issued the command 'rm -fr *' at the wrong point in the directory structure. Anyone who had files that were removable by this user was effected. Thankfully, we have a backup from before the command was run, a minority of users have overly permissive files, and a further minority were severely impacted. https://phabricator.wikimedia.org/T169774 was created in response to inquiries surrounding data loss.
We do not guarantee any level of user backups for day-to-day operations, but in this case since we do have the data I have restored it to /data/scratch/T169774/ so users can retrieve what was removed. We intend to make this restored data available until at least 2017-08-08. A Warning: Please do not rely on NFS for backups of code or critical data. We only have capacity to keep 2 weeks of historical backups at the moment and cannot guarantee timely retrieval or availability. Every Tool account can use https://phabricator.wikimedia.org/diffusion/ for code hosting, and creation of the repository is handled by going to https://toolsadmin.wikimedia.org/tools/id/<mytool>. This calamity was almost entirely caused by directories with o+w set allowing 'other' or 'everyone' write access. Do not use permissions such as '777' or that look like 'drwxrwxrwx' as it will allow other users to remove your files. This is especially dangerous in a shared hosting environment as this incident has shown. A brief explanation of why this happened to users who have given write permissions to 'other' for a directory in their Tool: *Because directories are not used in the same way as regular files, the permissions work slightly (but only slightly) differently. An attempt to list the files in a directory requires read permission for the directory, but not on the files within. An attempt to add a file to a directory, delete a file from a directory, or to rename a file, all require write permission for the directory, but (perhaps surprisingly) not for the files within. * *- *Unix File and Directory Permissions and Modes ( https://wpollock.com/AUnix1/FilePermissions.htm) ACTION ITEMS: - Make sure you have backups of code and data needed - Check for removed data you want to restore on login.tools.wmflabs.org at /data/scratch/T169774/ - Check your Tools files and directories for o+w permission and remove if possible (chmod -R o-w <directory>). - Ask for help on the labs-l mailing list, Phabricator, or in the #wikimedia-cloud IRC channel if you cannot figure out how to do without o+w (someone may have a different solution). -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC
_______________________________________________ Labs-announce mailing list labs-annou...@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-announce
_______________________________________________ Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l