Re: [BackupPC-users] Backuppc hangs on certain files.
Thanks Stefan Regards Brent On 31/05/2018 13:56, Stefan Peter wrote: Dear Brent Clark, On 31.05.2018 13:29, Brent Clark wrote: As you can see they are the same. 3.1.1-3+deb8u1 vs 3.1.1-3ubuntu1.2 They may have the same base version but they differ in distro specific patches. Anyway, the message 2018/05/31 08:59:52 [4524] ABORTING due to unsafe pathname from sender: /COU-IMMPRO.pdf seems to point to the rsyncd server dropping the connection. I'd investigate in this direction. With kind regards Stefan Peter -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on certain files.
Dear Brent Clark, On 31.05.2018 13:29, Brent Clark wrote: > As you can see they are the same. > > 3.1.1-3+deb8u1 vs 3.1.1-3ubuntu1.2 They may have the same base version but they differ in distro specific patches. Anyway, the message > 2018/05/31 08:59:52 [4524] ABORTING due to unsafe pathname from sender: > /COU-IMMPRO.pdf seems to point to the rsyncd server dropping the connection. I'd investigate in this direction. With kind regards Stefan Peter -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on certain files.
Good day Stefan Thank you for replying. That is very interesting, for I replied to Craig earlier, saying I successfully restored to a Debian machine. My problem machine is an Ubuntu 16.04 (xenial), but my test machine is Debian Jessie. When I saw the restore completed on the Debian machine this morning, I immediately checked the versions. As you can see they are the same. 3.1.1-3+deb8u1 vs 3.1.1-3ubuntu1.2 The Backuppc server is Debian stretch though. Regards Brent Clark On 31/05/2018 11:10, Stefan Peter wrote: Hi Brent On 31.05.2018 09:13, Brent Clark wrote: I too set rsyncd debugging (max verbosity = 2) and in the log I get: 2018/05/31 08:59:52 [4524] name lookup failed for REMOVED: Name or service not known 2018/05/31 08:59:52 [4524] connect from UNKNOWN (REMOVED) 2018/05/31 08:59:52 [4524] rsync to REMOVED/download/cougar/ from REMOVED@UNKNOWN (REMOVED) 2018/05/31 08:59:52 [4524] receiving file list 2018/05/31 08:59:52 [4524] ABORTING due to unsafe pathname from sender: /COU-IMMPRO.pdf Could it be that you are using Debian stretch and are bitten by this bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863334 Does the download/cougar directory exist on the system you want to restore to? With kind regards Stefan Peter -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on certain files.
Hi Brent On 31.05.2018 09:13, Brent Clark wrote: > > I too set rsyncd debugging (max verbosity = 2) and in the log I get: > > 2018/05/31 08:59:52 [4524] name lookup failed for REMOVED: Name or > service not known > 2018/05/31 08:59:52 [4524] connect from UNKNOWN (REMOVED) > 2018/05/31 08:59:52 [4524] rsync to REMOVED/download/cougar/ from > REMOVED@UNKNOWN (REMOVED) > 2018/05/31 08:59:52 [4524] receiving file list > 2018/05/31 08:59:52 [4524] ABORTING due to unsafe pathname from sender: > /COU-IMMPRO.pdf Could it be that you are using Debian stretch and are bitten by this bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863334 Does the download/cougar directory exist on the system you want to restore to? With kind regards Stefan Peter -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on certain files.
Good day Craig
This issue is weirder by the minute, but I thought I would keep you updated.
I quickly did the same restore (no config changes to backuppc) to a
Debian machine.
The rsyncd settings are the same on the restore host.
There were no issues restoring to the Debian machine.
So I am at a further loss.
Kind Regards
Brent Clark
On 31/05/2018 09:13, Brent Clark wrote:
Good day Craig
Thank you so much for taking the time to reply.
I hope I correctly did what you asked.
Im using Debians packaged version I.e.
# dpkg -l | grep backuppc
ii backuppc 3.3.1-4 amd64 high-performance,
enterprise-grade system for backing up PCs
I did what you asked.
I set the follopwing in backuppc config.pl as well as the hosts config
file (see attached).
$Conf{XferLogLevel} = '8';
$Conf{RsyncRestoreArgs} = '-vvv'
Here is the tail of the log:
Contents of file /var/lib/backuppc/pc/REMOVED/RestoreLOG.32.z,
modified 2018-05-31 08:59:54
Connected to REMOVED:873, remote version 31
Negotiated protocol version 28
Got response: fc1338e205d19c0e5239d2a203efcf84
in mime: /BM44gXRnA5SOdKiA+/PhA
Auth: got challenge: A6+NF2ig5v6hstnJy3QzMw, reply: REMOVED
/BM44gXRnA5SOdKiA+/PhA
Connected to module REMOVED
Sending args: --server -vvv . /REMOVED/download/cougar/
Checksum seed is 1527749992
Got checksumSeed 0x5b0f9d68
fileListSend: sending file list: REMOVED/download/cougar/COU-IMMPRO.pdf
Sending REMOVED/download/cougar/COU-IMMPRO.pdf
(remote=/COU-IMMPRO.pdf) type = 0
restore 664 10030/10030 37676705 REMOVED/cougar/COU-IMMPRO.pdf
Sorted file list has 1 entries
PostSortFile 0: /COU-IMMPRO.pdf
Read EOF:
Tried again: got 0 bytes
Done: 1 files, 37676705 bytes
restore failed: Unable to read 4 bytes
I too set rsyncd debugging (max verbosity = 2) and in the log I get:
2018/05/31 08:59:52 [4524] name lookup failed for REMOVED: Name or
service not known
2018/05/31 08:59:52 [4524] connect from UNKNOWN (REMOVED)
2018/05/31 08:59:52 [4524] rsync to REMOVED/download/cougar/ from
REMOVED@UNKNOWN (REMOVED)
2018/05/31 08:59:52 [4524] receiving file list
2018/05/31 08:59:52 [4524] ABORTING due to unsafe pathname from
sender: /COU-IMMPRO.pdf
2018/05/31 08:59:52 [4524] rsync error: protocol incompatibility (code
2) at flist.c(742) [Receiver=3.1.1]
Attached you will see a sanitised backuppc configuration for the host.
Thank you again for your help.
Regards
Brent
On 30/05/2018 19:47, Craig Barratt via BackupPC-users wrote:
-vvv to
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on certain files.
Good day Craig
Thank you so much for taking the time to reply.
I hope I correctly did what you asked.
Im using Debians packaged version I.e.
# dpkg -l | grep backuppc
ii backuppc 3.3.1-4 amd64
high-performance, enterprise-grade system for backing up PCs
I did what you asked.
I set the follopwing in backuppc config.pl as well as the hosts config
file (see attached).
$Conf{XferLogLevel} = '8';
$Conf{RsyncRestoreArgs} = '-vvv'
Here is the tail of the log:
Contents of file /var/lib/backuppc/pc/REMOVED/RestoreLOG.32.z, modified
2018-05-31 08:59:54
Connected to REMOVED:873, remote version 31
Negotiated protocol version 28
Got response: fc1338e205d19c0e5239d2a203efcf84
in mime: /BM44gXRnA5SOdKiA+/PhA
Auth: got challenge: A6+NF2ig5v6hstnJy3QzMw, reply: REMOVED
/BM44gXRnA5SOdKiA+/PhA
Connected to module REMOVED
Sending args: --server -vvv . /REMOVED/download/cougar/
Checksum seed is 1527749992
Got checksumSeed 0x5b0f9d68
fileListSend: sending file list: REMOVED/download/cougar/COU-IMMPRO.pdf
Sending REMOVED/download/cougar/COU-IMMPRO.pdf (remote=/COU-IMMPRO.pdf) type = 0
restore 664 10030/1003037676705 REMOVED/cougar/COU-IMMPRO.pdf
Sorted file list has 1 entries
PostSortFile 0: /COU-IMMPRO.pdf
Read EOF:
Tried again: got 0 bytes
Done: 1 files, 37676705 bytes
restore failed: Unable to read 4 bytes
I too set rsyncd debugging (max verbosity = 2) and in the log I get:
2018/05/31 08:59:52 [4524] name lookup failed for REMOVED: Name or service not
known
2018/05/31 08:59:52 [4524] connect from UNKNOWN (REMOVED)
2018/05/31 08:59:52 [4524] rsync to REMOVED/download/cougar/ from
REMOVED@UNKNOWN (REMOVED)
2018/05/31 08:59:52 [4524] receiving file list
2018/05/31 08:59:52 [4524] ABORTING due to unsafe pathname from sender:
/COU-IMMPRO.pdf
2018/05/31 08:59:52 [4524] rsync error: protocol incompatibility (code 2) at
flist.c(742) [Receiver=3.1.1]
Attached you will see a sanitised backuppc configuration for the host.
Thank you again for your help.
Regards
Brent
On 30/05/2018 19:47, Craig Barratt via BackupPC-users wrote:
-vvv to
# THIS FILE IS AUTOMATICALLY DISTRIBUTED BY PUPPET. ANY CHANGES WILL BE
# OVERWRITTEN.
$Conf{BackupFilesExclude} = {
'*' => [
'REMOVED_LIST',
]
};
$Conf{PingMaxMsec} = '200';
$Conf{IncrLevels} = [
'1',
'2',
'3',
'4',
'5',
'6'
];
$Conf{BlackoutPeriods} = [
{
'hourEnd' => '21.5',
'weekDays' => [
'1',
'2',
'3',
'4',
'5',
'6',
'7'
],
'hourBegin' => '8'
}
];
$Conf{RsyncShareName} = [
'REMOVED'
];
$Conf{RsyncArgs} = [
'--numeric-ids',
'--perms',
'--owner',
'--group',
'-D',
'--links',
'--hard-links',
'--times',
'--inplace',
'--recursive',];
$Conf{RsyncdPasswd} = 'REMOVED';
$Conf{XferLogLevel} = '8';
$Conf{RsyncRestoreArgs} = '-vvv'
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on certain files.
Brent,
First you should tell us which versions of rsync, BackupPC and rsync-bpc
you are using. If they are old, you should upgrade and try again.
Assuming you are using relatively recent versions (ie, at least 4.1.x), you
should run the restore with a high level of debug logging. Specifically:
- set $Conf{XferLogLevel} to 8
- increase the rsync logging by adding -vvv to $Conf{RsyncRestoreArgs}
- run a restore of that file, and send me the RestoreLOG file (don't
post to the list)
- then change the logging back, or else your next backup will generate a
huge log file.
Craig
On Wed, May 30, 2018 at 7:15 AM, Brent Clark
wrote:
> Good day Guys
>
> I am sitting with a very weird situation, and I am hoping someone can help.
>
> I needed to restore some files on an Ubuntu machine. Whats weird is,
> certain files just sit there and do not complete.
>
> e.g.
>
> lsof -ad3-999 -c rsync
>
> rsync 7372 root7u REG 252,7 37486592 3174459
> /REMOVED/download/cougar/.COU-IMMPRO.pdf.NYSypa
>
> I went to the directory and selected one file to be stored, and as said
> rsync / backuppc just sit there.
>
> The above is a 38M pdf. And in this case always this file, thats a problem.
>
> If I select all the others files for the download directory, then there is
> no problem with the restore .
>
> I have been doing strace on rsync and backuppc_restore. It just sits
> there. I.e. Nothing happening / happens.
>
> I was thinking it was Ubuntus apparmour that was stopping the renaming
> '.COU-IMMPRO.pdf.NYSypa' to the correct name.
>
> I would like to say, backups is no problem, its just restore.
>
> If anyone can assist, it would be gratefully appreciated.
>
> Kind Regards
>
> Brent Clark
>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] Backuppc hangs on certain files.
Good day Guys I am sitting with a very weird situation, and I am hoping someone can help. I needed to restore some files on an Ubuntu machine. Whats weird is, certain files just sit there and do not complete. e.g. lsof -ad3-999 -c rsync rsync 7372 root 7u REG 252,7 37486592 3174459 /REMOVED/download/cougar/.COU-IMMPRO.pdf.NYSypa I went to the directory and selected one file to be stored, and as said rsync / backuppc just sit there. The above is a 38M pdf. And in this case always this file, thats a problem. If I select all the others files for the download directory, then there is no problem with the restore . I have been doing strace on rsync and backuppc_restore. It just sits there. I.e. Nothing happening / happens. I was thinking it was Ubuntus apparmour that was stopping the renaming '.COU-IMMPRO.pdf.NYSypa' to the correct name. I would like to say, backups is no problem, its just restore. If anyone can assist, it would be gratefully appreciated. Kind Regards Brent Clark -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC hangs
Hi Holger,
thank you for your help. I used your commands to identify the files. I
had several chains with length of nearly 2000 files (propably from
rendered images for a movie in TIFF format, 16 MB size each), that made
all so slow. I zipped several directories with these images and backup
seems to run now.
Good to know that BackupPC is as solid as I thought, using it for many
many years now.
Thank you all!
Markus
Am 10.10.16 um 20:39 schrieb Holger Parplies:
> Hi,
>
> Markus Hirschmann wrote on 2016-10-10 17:54:45 +0200 [[BackupPC-users]
> BackupPC hangs]:
>>
>> I backup a huge QNAS Storage with rsync over ssh and it always hangs
>> with 100% load. I tried to find the problem with lsof:
>>
>> [...]
>> BackupPC_ 1842 backuppc6r REG 253,330938 1330743
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185
>
> 185 ... wow ...
>
>> BackupPC_ 1842 backuppc7r REG 253,377320 419524
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0
>> BackupPC_ 1842 backuppc9r REG 253,3 171056 419525
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1
>> BackupPC_ 1842 backuppc 11r REG 253,372265 419526
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2
>
> [and so on]
>
> You've got a rather long hash chain here (185 candidates for a pool match).
> For the current file in question, BackupPC needs to compare the contents with
> all candidates (or at least those that haven't been ruled out yet ... see
> the missing numbers further down:
>
>> BackupPC_ 1842 backuppc 27r REG 253,3 100064 419572
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18
>> BackupPC_ 1842 backuppc 28r REG 253,3 113544 419587
>> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33
>
> ). I'm guessing the file is rather large, so the comparison takes quite long.
> Note also that BackupPC won't take all candidates into account at once, due
> to the limit on concurrently open files, though I'm not sure what the strategy
> is (it might mean the comparison needs to be repeated).
>
> Hash collisions happen (mainly) for files with identical *length* and
> identical
> first and last 128 KiB chunk within the first 1 MiB of data (if I remember
> correctly; the details aren't important, so I won't check). Maybe you're
> backing up a large database with changes just in the wrong places? Log files
> would tend to grow, i.e. change length (and therefore hash), so they usually
> won't cause trouble in form of hash chains.
>
> Hint: 'ls -ali
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0'
> (or any other file in the chain) for a quick glance at (compressed) length and
> number of links (an extremely large number of copies would also tend to make
> hash chains longer than they need to be as well as make comparisons tedious -
> your $Conf{HardLinkMax} isn't by chance set to a ridiculously small value, is
> it?). You might also want to 'BackupPC_zcat !$ | wc -c' (please expand the
> !$ ;-) for an exact uncompressed file size. The inode number is included in
> the ls example, in case you feel like doing a 'find $TopDir/pc -inum ...' for
> locating the file(s) in question.
>
>> [...] I should delete my pool and start new but I don't want to do that. :/
>
> I'm not sure that would help much. If your backup strategy continues to backup
> this content, you'll probably run into the same situation again in the future.
> If you can and want to remove the file from (some) previous backups, search
> the list for ... err ... something like BackupPC_delete - a script to safely
> delete individual files from backups. BackupPC_nightly should take care of
> cleaning up (i.e. chain renumbering), *BUT* *PLEASE don't run it manually*!
>
> Of course, if you don't need to backup the file, simply excluding it would
> also fix things, as the comparison wouldn't happen in the future.
>
> Hope that helps.
>
> Regards,
> Holger
>
--
Markus Hirschmann, Dipl.-Inf. Univ.
Veitshöchheimer Straße 1, 97080 Würzburg
Tel: +49 931 8092864Mobil: +49 179 5408429
m...@durtro.deSteuernummer 257/229/10802
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC hangs
Hi,
Markus Hirschmann wrote on 2016-10-10 17:54:45 +0200 [[BackupPC-users] BackupPC
hangs]:
>
> I backup a huge QNAS Storage with rsync over ssh and it always hangs
> with 100% load. I tried to find the problem with lsof:
>
> [...]
> BackupPC_ 1842 backuppc6r REG 253,330938 1330743
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185
185 ... wow ...
> BackupPC_ 1842 backuppc7r REG 253,377320 419524
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0
> BackupPC_ 1842 backuppc9r REG 253,3 171056 419525
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1
> BackupPC_ 1842 backuppc 11r REG 253,372265 419526
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2
[and so on]
You've got a rather long hash chain here (185 candidates for a pool match).
For the current file in question, BackupPC needs to compare the contents with
all candidates (or at least those that haven't been ruled out yet ... see
the missing numbers further down:
> BackupPC_ 1842 backuppc 27r REG 253,3 100064 419572
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18
> BackupPC_ 1842 backuppc 28r REG 253,3 113544 419587
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33
). I'm guessing the file is rather large, so the comparison takes quite long.
Note also that BackupPC won't take all candidates into account at once, due
to the limit on concurrently open files, though I'm not sure what the strategy
is (it might mean the comparison needs to be repeated).
Hash collisions happen (mainly) for files with identical *length* and identical
first and last 128 KiB chunk within the first 1 MiB of data (if I remember
correctly; the details aren't important, so I won't check). Maybe you're
backing up a large database with changes just in the wrong places? Log files
would tend to grow, i.e. change length (and therefore hash), so they usually
won't cause trouble in form of hash chains.
Hint: 'ls -ali /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0'
(or any other file in the chain) for a quick glance at (compressed) length and
number of links (an extremely large number of copies would also tend to make
hash chains longer than they need to be as well as make comparisons tedious -
your $Conf{HardLinkMax} isn't by chance set to a ridiculously small value, is
it?). You might also want to 'BackupPC_zcat !$ | wc -c' (please expand the
!$ ;-) for an exact uncompressed file size. The inode number is included in
the ls example, in case you feel like doing a 'find $TopDir/pc -inum ...' for
locating the file(s) in question.
> [...] I should delete my pool and start new but I don't want to do that. :/
I'm not sure that would help much. If your backup strategy continues to backup
this content, you'll probably run into the same situation again in the future.
If you can and want to remove the file from (some) previous backups, search
the list for ... err ... something like BackupPC_delete - a script to safely
delete individual files from backups. BackupPC_nightly should take care of
cleaning up (i.e. chain renumbering), *BUT* *PLEASE don't run it manually*!
Of course, if you don't need to backup the file, simply excluding it would
also fix things, as the comparison wouldn't happen in the future.
Hope that helps.
Regards,
Holger
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] BackupPC hangs
Hi, I backup a huge QNAS Storage with rsync over ssh and it always hangs with 100% load. I tried to find the problem with lsof: BackupPC_ 1842 backuppc3w REG 253,3 2490 285576417 /var/lib/backuppc/pc/data/LOG.102016 BackupPC_ 1842 backuppc4w REG 253,3 4447002 285576424 /var/lib/backuppc/pc/data/XferLOG.z BackupPC_ 1842 backuppc5w REG 253,3 3130697 2324 /var/lib/backuppc/pc/data/NewFileList BackupPC_ 1842 backuppc6r REG 253,330938 1330743 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185 BackupPC_ 1842 backuppc7r REG 253,377320 419524 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0 BackupPC_ 1842 backuppc8u unix 0x880064e68dc0 0t0 997783 socket BackupPC_ 1842 backuppc9r REG 253,3 171056 419525 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1 BackupPC_ 1842 backuppc 10u unix 0x880064e69780 0t0 997109 socket BackupPC_ 1842 backuppc 11r REG 253,372265 419526 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2 BackupPC_ 1842 backuppc 12r REG 253,372310 419527 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_3 BackupPC_ 1842 backuppc 13r REG 253,372292 419538 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_4 BackupPC_ 1842 backuppc 14r REG 253,3 109717 419539 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_5 BackupPC_ 1842 backuppc 15r REG 253,3 150976 419540 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_6 BackupPC_ 1842 backuppc 16r REG 253,396457 419541 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_7 BackupPC_ 1842 backuppc 17r REG 253,396459 419542 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_8 BackupPC_ 1842 backuppc 18r REG 253,396699 419563 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_9 BackupPC_ 1842 backuppc 19r REG 253,397760 419564 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_10 BackupPC_ 1842 backuppc 20r REG 253,399200 419565 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_11 BackupPC_ 1842 backuppc 21r REG 253,3 101227 419566 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_12 BackupPC_ 1842 backuppc 22r REG 253,3 102926 419567 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_13 BackupPC_ 1842 backuppc 23r REG 253,3 103426 419568 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_14 BackupPC_ 1842 backuppc 24r REG 253,3 102631 419569 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_15 BackupPC_ 1842 backuppc 25r REG 253,3 101767 419570 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_16 BackupPC_ 1842 backuppc 26r REG 253,3 100944 419571 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_17 BackupPC_ 1842 backuppc 27r REG 253,3 100064 419572 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18 BackupPC_ 1842 backuppc 28r REG 253,3 113544 419587 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33 Nothing special in the logs. It counts the files up and up and then restarts with that. I should delete my pool and start new but I don't want to do that. :/ Any ideas? Thanks! Markus -- Markus Hirschmann, Dipl.-Inf. Univ. Veitshöchheimer Straße 1, 97080 Würzburg Tel: +49 931 8092864Mobil: +49 179 5408429 m...@durtro.de Steuernummer 257/229/10802 -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] backuppc hangs backing up lxc-host
Hello folks,
I've some trouble backing aa lxc container using rsync. The process
freezes - no error is reported so far.
stracing rsync on that machine I get:
[ ... lots of, lots of debug data ... ]
lstat(bin/pidof, {st_mode=S_IFLNK|0777, st_size=14, ...}) = 0
readlink(bin/pidof, /sbin/killall5..., 4095) = 14
lstat(bin/getfacl, {st_mode=S_IFREG|0755, st_size=23584, ...}) = 0
lstat(bin/lessecho, {st_mode=S_IFREG|0755, st_size=10344, ...}) = 0
lstat(bin/echo, {st_mode=S_IFREG|0755, st_size=27008, ...}) = 0
lstat(bin/stty, {st_mode=S_IFREG|0755, st_size=72224, ...}) = 0
lstat(bin/bzmore, {st_mode=S_IFREG|0755, st_size=1297, ...}) = 0
getdents(3, /* 0 entries */, 32768) = 0
close(3)= 0
open(selinux, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, /* 3 entries */, 32768) = 80
lstat(selinux/enforce, {st_mode=S_IFREG|0644, st_size=2, ...}) = 0
getdents(3, /* 0 entries */, 32768) = 0
close(3)= 0
mmap(NULL, 15245312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7f0633a66000
munmap(0x7f0633a66000, 15245312)= 0
select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left {59, 98})
write(1, \r\10\0\7, 4)= 4
select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left {59, 99})
write(1, \02085-hwclock.rules\253\0\0\0\316r\307P:\21\01660-g...,
2061) = 2061
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}
What may be wrong here?
Thanks,
Jan
--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] backuppc hangs when restoring folder with large amount of subdirs and content
Hello, I'm trying to restore a backup folder from an Ubuntu server to a windows 7 pc with cygwin/rsyncd the restore hangs. When I restore a subfolder with a small amount of content it recovers fine, but when I restore the entire folder it just hangs on the smallest file, there is no pattern and debugging is hard because it just hangs on a file, being busy but not failing. when i 'watch' lsof i can see the file it hangs at, that's it. Could it be because of filepaths that are to long because of the amount of subdirs? I hope someone can help! I know my backup is there, but restoring all the dirs separately is an impossible job! Arjen -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] backuppc hangs when restoring folder with large amount of subdirs and content
On 2015-03-03 09:07, Arjen Klaverstijn wrote: Hello, I'm trying to restore a backup folder from an Ubuntu server to a windows 7 pc with cygwin/rsyncd the restore hangs. When I restore a subfolder with a small amount of content it recovers fine, but when I restore the entire folder it just hangs on the smallest file, there is no pattern and debugging is hard because it just hangs on a file, being busy but not failing. when i 'watch' lsof i can see the file it hangs at, that's it. Could it be because of filepaths that are to long because of the amount of subdirs? I hope someone can help! I know my backup is there, but restoring all the dirs separately is an impossible job! Arjen One thing to keep in mind is that Windows file semantics will prevent restoration of open files; rsync will generally just wait for the file to no longer be open, which will never happen. You can always restore the files using some other method (zip, tar) if this is the case, but (of course) you won't be able to put them in place until the files are closed (this can be accomplished using the recovery console for files which are *never* closed, such as the registry. Since you're using lsof, it seems as if you're diagnosing the wrong end. What's happening at the Windows end? At any rate, other things that can cause this behavior: * Lack of permission * NTFS corruption * Junctions * Character encoding mismatches * Anti-virus software * buggy versions of rsync * not waiting long enough That's probably not an exhaustive list. Sure, lots of folders might make the list, but I'd think that would [also] cause a problem backing up the files in the first place. -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] backuppc hangs when rsync XferMethod is used for client with openssh-6.0
Hi, few days ago I've upgraded my box from openssh-5.9 to openssh-6.0 and since then I've seen many hangs from backuppc (actually no full backup was completed, e.g. still running after 24hours, usually it finishes in ~ 3 hours - 60GB) rsync --server was running fine (I was able to rsync whole disk with it) perl process was stuck in select() call: pid 5553: /usr/bin/perl /usr/bin/BackupPC_dump -v -f 127.0.0.1 # strace -p 5553 Process 5553 attached select(16, [10], NULL, [10], NULL^CProcess 5553 detached # gdb --pid=5553 (gdb) bt #0 0x7fb20ba3f6e3 in select () from /lib64/libc.so.6 #1 0x7fb20cb69465 in Perl_pp_sselect () from /usr/lib64/libperl.so.5.14 #2 0x7fb20cb1f306 in Perl_runops_standard () from /usr/lib64/libperl.so.5.14 #3 0x7fb20cac0d6a in perl_run () from /usr/lib64/libperl.so.5.14 #4 0x00400e89 in main () Usually it rsynced 24MB to new backup, sometimes more, but e.g. XferLogLevel and --verbose haven't changed that. Looking at openssh changelog I suspect this 2 changes: https://bugzilla.mindrot.org/show_bug.cgi?id=1859 https://bugzilla.mindrot.org/show_bug.cgi?id=1943 but it could be something completly different, but now after downgrade back to 5.9 I've rsynced 2,7G to new sofar and it's still running. I'll try to narrow this a bit more (even bisect openssh if needed), but wanted to report it here asap so people don't need to debug perl and other stuff like I did (not expecting openssh to be the cause). Or maybe someone already knows about better work around with -W or something which should be added to default BackupPC config.. Cheers, -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on backing up WinXP c-drive
On Sun, Oct 19, 2008 at 8:49 AM, Jeffrey J. Kosowsky [EMAIL PROTECTED] wrote: I am using the rsync-method to back up the C-drive on a WinXP machine. I have cygwin ssh rsync installed on the remote machine. The backup repeatedly stalls after backing up: .file_store_32/runescape/main_file_cache.dat1 and in the middle (presumably) of backing up: .file_store_32/runescape/main_file_cache.dat2 This occurs only about a dozen files into the backup. I found this to be a bug with rsync launched from ssh. Even outside of Backuppc we consistently found problems with rsync tunneled inside of ssh. We would've liked to use ssh to encrypt all of our backups, but ended up going with the straight rsyncd instead, which has worked great. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backuppc hangs on backing up WinXP c-drive
Linux Punk wrote at about 16:53:09 -0600 on Wednesday, October 22, 2008: On Sun, Oct 19, 2008 at 8:49 AM, Jeffrey J. Kosowsky [EMAIL PROTECTED] wrote: I am using the rsync-method to back up the C-drive on a WinXP machine. I have cygwin ssh rsync installed on the remote machine. The backup repeatedly stalls after backing up: .file_store_32/runescape/main_file_cache.dat1 and in the middle (presumably) of backing up: .file_store_32/runescape/main_file_cache.dat2 This occurs only about a dozen files into the backup. I found this to be a bug with rsync launched from ssh. Even outside of Backuppc we consistently found problems with rsync tunneled inside of ssh. We would've liked to use ssh to encrypt all of our backups, but ended up going with the straight rsyncd instead, which has worked great. I guess what I don't understand is why it works when I manually run rsync over ssh. The only difference that I can see is that when I run it manually I don't use the --server --sender flags but instead use rsync root@target. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Backuppc hangs on backing up WinXP c-drive
I am using the rsync-method to back up the C-drive on a WinXP machine. I have cygwin ssh rsync installed on the remote machine. The backup repeatedly stalls after backing up: .file_store_32/runescape/main_file_cache.dat1 and in the middle (presumably) of backing up: .file_store_32/runescape/main_file_cache.dat2 This occurs only about a dozen files into the backup. The verbose output is as follows: full backup started for directory /c/ started full dump, share=/c/ Running: /usr/bin/ssh -q -x -l kosowsky mywinxppc /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --one-file-system --omit-dir-times --ignore-times . /c/ Xfer PIDs are now 24412 xferPids 24412 Got remote protocol 29pp Negotiated protocol version 28 Checksum caching enabled (checksumSeed = 32761) Sent exclude: /Documents and Settings/*/Local Settings/Temp/* Sent exclude: /Documents and Settings/*/Local Settings/Temporary Internet Files/* Sent exclude: /System Volume Information Sent exclude: /pagefile.sys Sent exclude: /hiberfil.sys Sent exclude: /RECYCLER Remote[2]: file has vanished: /c/Documents and Settings/User1/My Documents/My Music/iTunes/iTunes Music/ 7 - ??? ?? ??? Remote[1]: rsync: opendir /c/Documents and Settings/User2/Application Data/Thunderbird/Profiles/9n4fftcz.default/Mail/User3 failed: No such file or directory (2) Remote[2]: file has vanished: /c/Documents and Settings/User2/Favorites/Computer/broadband + Forums + Linksys + BEFSR41.url Remote[2]: file has vanished: /c/Documents and Settings/User4/Application Data/Microsoft/Office/Recent/? ?? ?'.LNK (plus about 20 more messages like the above) Xfer PIDs are now 24412,24448 xferPids 24412,24448 create d 775 544/544 0 . create d 700 1010/513 0 .file_store_32 create 700 1010/513 211957 .file_store_32/code.dat create 700 1010/513 768989 .file_store_32/main_file_cache.dat create 700 1010/513 54 .file_store_32/main_file_cache.idx0 create 700 1010/513 0 .file_store_32/main_file_cache.idx1 create 700 1010/513 0 .file_store_32/main_file_cache.idx2 create 700 1010/513 0 .file_store_32/main_file_cache.idx3 create 700 1010/513 0 .file_store_32/main_file_cache.idx4 create 700 1009/513 61436 .file_store_32/music0.mid create d 700 1009/513 0 .file_store_32/runescape create 700 1009/513 209734 .file_store_32/runescape/main_file_cache.dat1 I have no problem using plain vanilla 'rsync -auxH mywinxppc:/c/ . to backup the files. Again it is frustrating that Backuppc seems to be so fragile in that it just seemingly hangs when it gets stuck on a file failing to either give any error messages (unless you manually run BackupPC_dump) or to explain what is wrong or to just skip the offending files and move on. I am using BackupPC-3.1.0 with rsync-2.6.9 The cygwin remote versions are: OpenSSH_4.5p1, OpenSSL 0.9.8d 28 Sep 2006 rsync version 2.6.6 protocol version 29 - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
