Hi Holger, thank you for your help. I used your commands to identify the files. I had several chains with length of nearly 2000 files (propably from rendered images for a movie in TIFF format, 16 MB size each), that made all so slow. I zipped several directories with these images and backup seems to run now.
Good to know that BackupPC is as solid as I thought, using it for many many years now. Thank you all! Markus Am 10.10.16 um 20:39 schrieb Holger Parplies: > Hi, > > Markus Hirschmann wrote on 2016-10-10 17:54:45 +0200 [[BackupPC-users] > BackupPC hangs]: >> >> I backup a huge QNAS Storage with rsync over ssh and it always hangs >> with 100% load. I tried to find the problem with lsof: >> >> [...] >> BackupPC_ 1842 backuppc 6r REG 253,3 30938 1330743 >> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185 > > 185 ... wow ... > >> BackupPC_ 1842 backuppc 7r REG 253,3 77320 419524 >> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0 >> BackupPC_ 1842 backuppc 9r REG 253,3 171056 419525 >> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1 >> BackupPC_ 1842 backuppc 11r REG 253,3 72265 419526 >> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2 > > [and so on] > > You've got a rather long hash chain here (185 candidates for a pool match). > For the current file in question, BackupPC needs to compare the contents with > all candidates (or at least those that haven't been ruled out yet ... see > the missing numbers further down: > >> BackupPC_ 1842 backuppc 27r REG 253,3 100064 419572 >> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18 >> BackupPC_ 1842 backuppc 28r REG 253,3 113544 419587 >> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33 > > ). I'm guessing the file is rather large, so the comparison takes quite long. > Note also that BackupPC won't take all candidates into account at once, due > to the limit on concurrently open files, though I'm not sure what the strategy > is (it might mean the comparison needs to be repeated). > > Hash collisions happen (mainly) for files with identical *length* and > identical > first and last 128 KiB chunk within the first 1 MiB of data (if I remember > correctly; the details aren't important, so I won't check). Maybe you're > backing up a large database with changes just in the wrong places? Log files > would tend to grow, i.e. change length (and therefore hash), so they usually > won't cause trouble in form of hash chains. > > Hint: 'ls -ali > /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0' > (or any other file in the chain) for a quick glance at (compressed) length and > number of links (an extremely large number of copies would also tend to make > hash chains longer than they need to be as well as make comparisons tedious - > your $Conf{HardLinkMax} isn't by chance set to a ridiculously small value, is > it?). You might also want to 'BackupPC_zcat !$ | wc -c' (please expand the > !$ ;-) for an exact uncompressed file size. The inode number is included in > the ls example, in case you feel like doing a 'find $TopDir/pc -inum ...' for > locating the file(s) in question. > >> [...] I should delete my pool and start new but I don't want to do that. :/ > > I'm not sure that would help much. If your backup strategy continues to backup > this content, you'll probably run into the same situation again in the future. > If you can and want to remove the file from (some) previous backups, search > the list for ... err ... something like BackupPC_delete - a script to safely > delete individual files from backups. BackupPC_nightly should take care of > cleaning up (i.e. chain renumbering), *BUT* *PLEASE don't run it manually*! > > Of course, if you don't need to backup the file, simply excluding it would > also fix things, as the comparison wouldn't happen in the future. > > Hope that helps. > > Regards, > Holger > -- Markus Hirschmann, Dipl.-Inf. Univ. Veitshöchheimer Straße 1, 97080 Würzburg Tel: +49 931 8092864 Mobil: +49 179 5408429 m...@durtro.de Steuernummer 257/229/10802 ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/