[Linux-users] delete huge nest of folders
Hi, over the holiday period some weirdness occurred and rectifying is proving troublesome. On a USB drive I have a very deeply nested folder structure of the same repeating name which probably ultimately ends in the same name and has a few 100MB of photos in it i.e. /media/drive/2003/2003/2003 (repeat many MANY times) /2003/somephotos.jpg. Attempting to do rm -rf may be doing something (shows in htop, taking a few % of a core) but a few hours later is still not done. Looking at the properties of the top level folder in a file manager, it reads for ages trying to determine number of folders and gets to a brick wall at about 140,000 folders and kills my KDE session! Some googling isn't really getting me anywhere, ideas like this do not finish the job after a couple of hours: ls | grep 2003 | xargs rm -rf Any suggestions on how to remove the offending folders in some quick-ish and recursive fashion are welcomed, while leaving other folders in /media alone. Secondary importance is ideas on how the situation arose in the first place might be interesting. Possible relevant factors: a 1TB USB drive encrypted via TrueCrypt receiving backups via rsync, the host machine had a power failure on the boxing day quake such that the power cable to one of the internal hard drives (containing /home) came out and left the machine in a partially useable state. Other top level folders on the same USB drive are fine. Cheers, Roger ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
Roger Searle wrote, On 01/17/2011 01:27 PM: Hi, over the holiday period some weirdness occurred and rectifying is proving troublesome. On a USB drive I have a very deeply nested folder structure of the same repeating name which probably ultimately ends in the same name and has a few 100MB of photos in it i.e. /media/drive/2003/2003/2003 (repeat many MANY times) /2003/somephotos.jpg. Attempting to do rm -rf may be doing something (shows in htop, taking a few % of a core) but a few hours later is still not done. Looking at the properties of the top level folder in a file manager, it reads for ages trying to determine number of folders and gets to a brick wall at about 140,000 folders and kills my KDE session! Some googling isn't really getting me anywhere, ideas like this do not finish the job after a couple of hours: ls | grep 2003 | xargs rm -rf Any suggestions on how to remove the offending folders in some quick-ish and recursive fashion are welcomed, while leaving other folders in /media alone. Copy your stuff to somewhere else. It might be unneeded, but better than losing it. Then I'd suggest you do a simple fsck and see if its upset. I've seen this sort of thing happen in FAT disks in the past. Given that you can't descend to the last level, its probably not worth trying to delete the directories. -- Craig Falconer ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
On Mon, Jan 17, 2011 at 1:27 PM, Roger Searle ro...@stepahead.org.nz wrote: /media/drive/2003/2003/2003 (repeat many MANY times) /2003/somephotos.jpg. Attempting to do rm -rf may be doing something (shows in htop, taking a few % of a core) but a few hours later is still not done. Looking at the properties of the top level folder in a file manager, it reads for ages trying to determine number of folders and gets to a brick wall at about 140,000 folders and kills my KDE session! So you have a structure of many directories, and at the bottom a set of files? Or something different? I presume you've removed the files at the bottom OK? Any suggestions on how to remove the offending folders in some quick-ish and recursive fashion are welcomed, while leaving other folders in /media alone. Errm, unlink() the top one (/media/drive/2003/2003 ?) using perl or C or something that isn't the shell, then unmount and fsck the filesystem :-) Then go remove anything left in the /media/drive/lost+found directory ... That's a little dangerous though, but if you like living on the edge, go for it. Also, given that this is a backup drive, don't bother preserving the rest of /media/drive, consider nuking it completely with mkfs, and then re-establishing the backups. Not sure what caused your problem, but I like to leave marker files on my mount pounts, where they exist when a device is unmounted (i.e. not /media/...) So when my drive is unmounted I might have /data -- so in there I'll create a file called /data/STOP. When the correct device is mounted onto /data, this file effectively disappears -- so I can check for it in the top of any simple backup scripts ... if [ -e /data/STOP ] then exit 1 fi This helps to detect unmounted directories, without having to know anything about how they are mounted in the script. -jim ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
Jim Cheetham wrote, On 01/17/2011 02:02 PM: Not sure what caused your problem, but I like to leave marker files on my mount pounts, where they exist when a device is unmounted (i.e. not /media/...) So when my drive is unmounted I might have /data -- so in there I'll create a file called /data/STOP. When the correct device is mounted onto /data, this file effectively disappears -- so I can check for it in the top of any simple backup scripts ... if [ -e /data/STOP ] then exit 1 fi This helps to detect unmounted directories, without having to know anything about how they are mounted in the script. That's a bloody good idea - I like it! We use automount to do this - it automatically mounts paths when they're used.Most distros package it as autofs At home I automount NFS shares, and at work its also used for USB drives for backups. -- Craig Falconer ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
On Mon, Jan 17, 2011 at 2:05 PM, C. Falconer cfalco...@totalteam.co.nz wrote: Jim Cheetham wrote, On 01/17/2011 02:02 PM: create a file called /data/STOP. When the correct device is mounted At home I automount NFS shares, and at work its also used for USB drives for backups. I've spent several months fighting with my mythtv backend using NFS to mount a remote machine's disk, so I can copy files to the TV playback machine. Sadly, when the NFS mount goes stale, I cannot fix it without a reboot of the mythbox ... which currently is also my router. So as an addition to /data/STOP, I also include /data/GO -- if I cannot read the contents of /data/GO, the mount must be stale. I've since replaced that playback box with a real mythtv frontend. Not sure where I'll use the disk next. -jim ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
Thanks Jim and Craig for the suggestion to do fsck and possibly write off the partition, as noted it is a backup drive and 1 of 2 that are regularly rotated because you never know when something might go wrong! I expect (and will check) that my rsyncs will have the correct files copied over next time I use that drive, it will just be a longer than normal sync. Not that it matters because it will be out of hours anyway. So I'll consider the problem solved. Cheers, Roger On 17/01/11 14:02, Jim Cheetham wrote: On Mon, Jan 17, 2011 at 1:27 PM, Roger Searlero...@stepahead.org.nz wrote: /media/drive/2003/2003/2003 (repeat many MANY times) /2003/somephotos.jpg. Attempting to do rm -rf may be doing something (shows in htop, taking a few % of a core) but a few hours later is still not done. Looking at the properties of the top level folder in a file manager, it reads for ages trying to determine number of folders and gets to a brick wall at about 140,000 folders and kills my KDE session! So you have a structure of many directories, and at the bottom a set of files? Or something different? I presume you've removed the files at the bottom OK? Any suggestions on how to remove the offending folders in some quick-ish and recursive fashion are welcomed, while leaving other folders in /media alone. Errm, unlink() the top one (/media/drive/2003/2003 ?) using perl or C or something that isn't the shell, then unmount and fsck the filesystem :-) Then go remove anything left in the /media/drive/lost+found directory ... That's a little dangerous though, but if you like living on the edge, go for it. Also, given that this is a backup drive, don't bother preserving the rest of /media/drive, consider nuking it completely with mkfs, and then re-establishing the backups. Not sure what caused your problem, but I like to leave marker files on my mount pounts, where they exist when a device is unmounted (i.e. not /media/...) So when my drive is unmounted I might have /data -- so in there I'll create a file called /data/STOP. When the correct device is mounted onto /data, this file effectively disappears -- so I can check for it in the top of any simple backup scripts ... if [ -e /data/STOP ] then exit 1 fi This helps to detect unmounted directories, without having to know anything about how they are mounted in the script. -jim ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
On Mon, Jan 17, 2011 at 2:44 PM, Roger Searle ro...@stepahead.org.nz wrote: Thanks Jim and Craig for the suggestion to do fsck and possibly write off the partition, as noted it is a backup drive and 1 of 2 that are regularly rotated because you never know when something might go wrong! Well, if you are writing it off, how about trying the unlink/fsck trick and see if that helps? I'd be keen to hear from someone using it in practice ... -jim ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
On 17/01/11 14:02, Jim Cheetham wrote: Not sure what caused your problem, but I like to leave marker files on my mount pounts, where they exist when a device is unmounted (i.e. not /media/...) So when my drive is unmounted I might have /data -- so in there I'll create a file called /data/STOP. When the correct device is mounted onto /data, this file effectively disappears -- so I can check for it in the top of any simple backup scripts ... if [ -e /data/STOP ] then exit 1 fi This helps to detect unmounted directories, without having to know anything about how they are mounted in the script. -jim This is excellent, I'll be making use of this method! Thanks, Roger ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
Re: [Linux-users] delete huge nest of folders
On 17/01/11 14:56, Jim Cheetham wrote: On Mon, Jan 17, 2011 at 2:44 PM, Roger Searlero...@stepahead.org.nz wrote: Thanks Jim and Craig for the suggestion to do fsck and possibly write off the partition, as noted it is a backup drive and 1 of 2 that are regularly rotated because you never know when something might go wrong! Well, if you are writing it off, how about trying the unlink/fsck trick and see if that helps? I'd be keen to hear from someone using it in practice ... -jim OK - but I got kind of stuck, and time pressures took me right to write it off. The drive must be mounted in TrueCrypt to be able to write to it, and while it is I got: roger@mercury:/usr/share/man$ fsck /media/truecrypt1/ fsck from util-linux-ng 2.17.2 e2fsck 1.41.11 (14-Mar-2010) fsck.ext2: Is a directory while trying to open /media/truecrypt1 The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 device So first question: Does fsck correctly know it's a corrupt superblock given it's encrypted? If I then unmount the drive in TrueCrypt, it does not show in the results of the mount command. This being the point at which I decided to move on! I've not yet done a mkfs. How would I then run fsck without a TrueCrypt mount? Or is the unlink suggestion done via other means? Cheers, Roger ___ Linux-users mailing list Linux-users@lists.canterbury.ac.nz http://lists.canterbury.ac.nz/mailman/listinfo/linux-users
