friedmann wrote at about 06:19:35 -0700 on Tuesday, August 9, 2011: > > I also noticed a good clue, pointing out the number of subdirectories > hence, i have counted the number of subdirectories that are on the backuped > drive, using : "find . -type d | wc -l", that returned me : > 82779
Well this by itself doesn't mean anything since find looks at the entire tree of subdirectories not just one level.... (you would need to use maxdepth to limit the find) > > And, maybe more insightful, i have 32002 subdirectories in the same > directory, i.e. in the "most populated" one (the one on which the > backup process fails : > /var/lib/backuppc/pc/192.168.2.105/new//f%2fmedia%2fRhenovia/fVBox_Data_until_January_2011/fSBML_221010_IP3R_ForArnaud/fsbml2java_distrib/fDATE27octobre2010TIME09h17_huge9) That is the problem since as you tested, the number of links on your system is limited to 32000 so you can't have more than that number of subdirectories either. > > I found out that, using ext3, a limit for the number of > subdirectories in one directory could be 31998 > (http://superuser.com/questions/66331/what-is-the-maximum-number-of-folders-allowed-in-a-folder-in-linux). I posted that limitation yesterday in response to your OP. > I checked this with a shell script the following shell script : > SCRIPT > #!/bin/bash > for i in {1..32010} > do > mkdir test$i > done > END SCRIPT > > It actually fails after 31998 created subdirectories on the ext3 partition > tested on the host's ntfs partition : the test is passed. I even tried with > a value of 500000 : I stopped it after 296000 subdirectories were created. > > [Idea] So i consider the problem as identified : It is relatively not > related to backuppc itself, but deals with a sort of filesystems > incompatibility. [Idea] Well it seems to me it is not just relatively but not at all related to backuppc. > [Question] And finally, the only question that lasts is : how can i increase > the number of subdirectories tolerated by ext3 ? [Question] You really can't/shouldn't unless you recompile your own custom kernel with a different hard-coded value, but even so I would strongly discourage doing so for the following reasons: 1. There is no guarantee that the resulting filesystem would be stable or efficient since it likely hasn't been (extensively) tested with other values. In particular, using extra bits to store or count links in something as low as a filesystem structure could cause overflow issues... 2. Your non-standard ext3 filesystem would be incompatible with normal implementations and could cause problems, including corruption, if you ever mistakenly mounted it on a normal version. I think there are only 2 reasonable atlernatives: 1. Use another filesystem to store your BackupPC data -- as was pointed out in the thread ext4 typically seems to have a 65000 limit while other filesystems like zfs have no such limit. 2. Break up the offending subdirectories on your client machines into another level of subdirectories so that the number of subdirectories is below HardLinkMax > Admit that it was quite tricky : eveything deals with a limit of 32000 > (hardlinks, subdirectories ...) :o > Not sure what was so tricky. It seemed pretty easy to figure out given that you provided essentially no specific debugging information yet the responders to your thread suggested that (1) issue is in rsync (2) might be aa subdirectory limitation ------------------------------------------------------------------------------ uberSVN's rich system and user administration capabilities and model configuration take the hassle out of deploying and managing Subversion and the tools developers use with it. Learn more about uberSVN and get a free download at: http://p.sf.net/sfu/wandisco-dev2dev _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
