Hi Holger,

thank you for your help. I used your commands to identify the files. I 
had several chains with length of nearly 2000 files (propably from 
rendered images for a movie in TIFF format, 16 MB size each), that made 
all so slow. I zipped several directories with these images and backup 
seems to run now.

Good to know that BackupPC is as solid as I thought, using it for many 
many years now.

Thank you all!


Am 10.10.16 um 20:39 schrieb Holger Parplies:
> Hi,
> Markus Hirschmann wrote on 2016-10-10 17:54:45 +0200 [[BackupPC-users] 
> BackupPC hangs]:
>> I backup a huge QNAS Storage with rsync over ssh and it always hangs
>> with 100% load. I tried to find the problem with lsof:
>> [...]
>> BackupPC_ 1842 backuppc    6r   REG              253,3    30938 1330743 
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185
> 185 ... wow ...
>> BackupPC_ 1842 backuppc    7r   REG              253,3    77320 419524 
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0
>> BackupPC_ 1842 backuppc    9r   REG              253,3   171056 419525 
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1
>> BackupPC_ 1842 backuppc   11r   REG              253,3    72265 419526 
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2
> [and so on]
> You've got a rather long hash chain here (185 candidates for a pool match).
> For the current file in question, BackupPC needs to compare the contents with
> all candidates (or at least those that haven't been ruled out yet ... see
> the missing numbers further down:
>> BackupPC_ 1842 backuppc   27r   REG              253,3   100064 419572 
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18
>> BackupPC_ 1842 backuppc   28r   REG              253,3   113544 419587 
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33
> ). I'm guessing the file is rather large, so the comparison takes quite long.
> Note also that BackupPC won't take all candidates into account at once, due
> to the limit on concurrently open files, though I'm not sure what the strategy
> is (it might mean the comparison needs to be repeated).
> Hash collisions happen (mainly) for files with identical *length* and 
> identical
> first and last 128 KiB chunk within the first 1 MiB of data (if I remember
> correctly; the details aren't important, so I won't check). Maybe you're
> backing up a large database with changes just in the wrong places? Log files
> would tend to grow, i.e. change length (and therefore hash), so they usually
> won't cause trouble in form of hash chains.
> Hint: 'ls -ali 
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0'
> (or any other file in the chain) for a quick glance at (compressed) length and
> number of links (an extremely large number of copies would also tend to make
> hash chains longer than they need to be as well as make comparisons tedious -
> your $Conf{HardLinkMax} isn't by chance set to a ridiculously small value, is
> it?). You might also want to 'BackupPC_zcat !$ | wc -c' (please expand the
> !$ ;-) for an exact uncompressed file size. The inode number is included in
> the ls example, in case you feel like doing a 'find $TopDir/pc -inum ...' for
> locating the file(s) in question.
>> [...] I should delete my pool and start new but I don't want to do that. :/
> I'm not sure that would help much. If your backup strategy continues to backup
> this content, you'll probably run into the same situation again in the future.
> If you can and want to remove the file from (some) previous backups, search
> the list for ... err ... something like BackupPC_delete - a script to safely
> delete individual files from backups. BackupPC_nightly should take care of
> cleaning up (i.e. chain renumbering), *BUT* *PLEASE don't run it manually*!
> Of course, if you don't need to backup the file, simply excluding it would
> also fix things, as the comparison wouldn't happen in the future.
> Hope that helps.
> Regards,
> Holger

Markus Hirschmann, Dipl.-Inf. Univ.
Veitshöchheimer Straße 1, 97080 Würzburg
Tel: +49 931 8092864    Mobil: +49 179 5408429
m...@durtro.de        Steuernummer 257/229/10802

Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
BackupPC-users mailing list
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Reply via email to