Hi Holger,
thank you for your help. I used your commands to identify the files. I
had several chains with length of nearly 2000 files (propably from
rendered images for a movie in TIFF format, 16 MB size each), that made
all so slow. I zipped several directories with these images and backup
seems to run now.
Good to know that BackupPC is as solid as I thought, using it for many
many years now.
Thank you all!
Markus
Am 10.10.16 um 20:39 schrieb Holger Parplies:
> Hi,
>
> Markus Hirschmann wrote on 2016-10-10 17:54:45 +0200 [[BackupPC-users]
> BackupPC hangs]:
>>
>> I backup a huge QNAS Storage with rsync over ssh and it always hangs
>> with 100% load. I tried to find the problem with lsof:
>>
>> [...]
>> BackupPC_ 1842 backuppc 6r REG 253,3 30938 1330743
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185
>
> 185 ... wow ...
>
>> BackupPC_ 1842 backuppc 7r REG 253,3 77320 419524
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0
>> BackupPC_ 1842 backuppc 9r REG 253,3 171056 419525
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1
>> BackupPC_ 1842 backuppc 11r REG 253,3 72265 419526
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2
>
> [and so on]
>
> You've got a rather long hash chain here (185 candidates for a pool match).
> For the current file in question, BackupPC needs to compare the contents with
> all candidates (or at least those that haven't been ruled out yet ... see
> the missing numbers further down:
>
>> BackupPC_ 1842 backuppc 27r REG 253,3 100064 419572
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18
>> BackupPC_ 1842 backuppc 28r REG 253,3 113544 419587
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33
>
> ). I'm guessing the file is rather large, so the comparison takes quite long.
> Note also that BackupPC won't take all candidates into account at once, due
> to the limit on concurrently open files, though I'm not sure what the strategy
> is (it might mean the comparison needs to be repeated).
>
> Hash collisions happen (mainly) for files with identical *length* and
> identical
> first and last 128 KiB chunk within the first 1 MiB of data (if I remember
> correctly; the details aren't important, so I won't check). Maybe you're
> backing up a large database with changes just in the wrong places? Log files
> would tend to grow, i.e. change length (and therefore hash), so they usually
> won't cause trouble in form of hash chains.
>
> Hint: 'ls -ali
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0'
> (or any other file in the chain) for a quick glance at (compressed) length and
> number of links (an extremely large number of copies would also tend to make
> hash chains longer than they need to be as well as make comparisons tedious -
> your $Conf{HardLinkMax} isn't by chance set to a ridiculously small value, is
> it?). You might also want to 'BackupPC_zcat !$ | wc -c' (please expand the
> !$ ;-) for an exact uncompressed file size. The inode number is included in
> the ls example, in case you feel like doing a 'find $TopDir/pc -inum ...' for
> locating the file(s) in question.
>
>> [...] I should delete my pool and start new but I don't want to do that. :/
>
> I'm not sure that would help much. If your backup strategy continues to backup
> this content, you'll probably run into the same situation again in the future.
> If you can and want to remove the file from (some) previous backups, search
> the list for ... err ... something like BackupPC_delete - a script to safely
> delete individual files from backups. BackupPC_nightly should take care of
> cleaning up (i.e. chain renumbering), *BUT* *PLEASE don't run it manually*!
>
> Of course, if you don't need to backup the file, simply excluding it would
> also fix things, as the comparison wouldn't happen in the future.
>
> Hope that helps.
>
> Regards,
> Holger
>
--
Markus Hirschmann, Dipl.-Inf. Univ.
Veitshöchheimer Straße 1, 97080 Würzburg
Tel: +49 931 8092864 Mobil: +49 179 5408429
m...@durtro.de Steuernummer 257/229/10802
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
