Re: [BackupPC-users] Avoiding long backup times
Hello all, I think this post by Holger is a pretty good explanation of the differences between the backup types and transfer methods available in BackupPC. Maybe this information could be reworked for inclusion in the BackupPC documentation? Nils Breunese. Holger Parplies wrote: Hi, Clemens von Musil wrote on 20.01.2007 at 00:31:12 [[BackupPC-users] Avoiding long backup times]: [...] One configured backup host has a very slow network connection - a full backup lasts about two days. Because of the exclusive run of backuppc_nightly in 2.xx, I learned, this full backup stalls all other backups. I want to avoid tihs situation and got stuck in following: What exactly happens during full backup? that depends on the transfer method. For a slow connection, you want rsync (or rsyncd), not tar or smb. For a very slow connection, you *definitely* want rsync/rsyncd (which I'll just call rsync for simplicity). I read, that backuppc stores every identical file only one time. Basically true, but BackupPC needs to determine that the file is identical to something and to what. Sparing you the transfer (if possible with reasonable cost) is rsyncs job. Doing so comes at a cost in terms of CPU usage, so you've got the option of using tar if bandwidth is cheaper than CPU power. In your case, it obviously is not. [...] What happens with an unchanged file in a full backup? tar and smb will transfer it, rsync will not (presuming you mean unchanged as in same name, same content). For a changed file, rsync will try to speed up the transfer. If you append a few bytes to a large file, tar/smb will transfer the whole file (even on an incremental), while rsync will (basically) transfer some checksums and the few bytes only (on full and incremental backups). If the file will not be transferred again - what is the difference between full and incremenal? This only applies to rsync (as tar/smb will transfer it). rsync will always transfer missing files as well as update files that have apparently changed. The difference between full and incremental backups lies firstly in the amount of trouble rsync will go to to determine whether a file has changed or not. For an incremental backup, I believe rsync will only look at size and modification time, whereas for a full backup, checksums of the files are calculated, thus consuming much more CPU-time and disk-I/O-bandwidth at least on the client (I understand the server caches checksums if you tell it to), with the benefit of detecting files that have been changed and the modification time reset. The second difference is that a full backup gives a new reference point for following backups while an incremental backup does not necessarily. In version 2 (of BackupPC) all incrementals were relative to the last full (so after one year of only incrementals you'd be transfering everything changed in that year on each incremental backup), whereas version 3 supports multi-level incrementals, with each incremental transfering everything changed since the last incremental of lower level (resp. full backup at level 0). And if yes - does it make sense to keep one full for ever and dealing only with following incrementals? No. With a very slow network connection, you want to avoid transfering changes more often than necessary. Configuring multi-level incrementals in BackupPC 3 seems to be simple enough that you *could* say do incrementals of increasing level each day for the next, say, 10 years, but that will make browsing, restoring and even doing the backups increasingly expensive (and you'll need to keep all the incrementals), even if you *can* neglect modified files your backups are missing. A full backup each day *might* be your best choice, because it avoids duplicating transfers. With any incrementals in between, you'll have at least the fulls re-transfering everything since the last full (less or equal to the sum of the intervening incrementals (plus the changes since the last incremental, obviously), due to files modified more than once or modified and deleted). You'll have to find out the best trade-off in your specific situation between retransfering file content and calculating checksums. You probably need to find out first, how much data you need to transfer each day. Then you can estimate, the traffic of how many days you could allow yourself to transfer at once. That should be about the maximum interval between full backups ... Regards, Holger -- --- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php? page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list
[BackupPC-users] trashClean start time seems wrong
Hello, I have been running BackupPC 2.1.2pl2 for several months now without a hitch. Just now I logged into the web interface and on the status page I see two currently running jobs. One is an incremental backup that started an hour ago and the other is a trashClean job that according to the status page was started on 15 december at 12:15. I'm pretty sure that when I logged in earlier today that job wasn't running yet, so I guess that start time is just wrong. Also, the trashClean start time is exactly the same date and time that the backuppc service was last restarted. Seems like there's a bug in displaying the correct start time? Other than that, all seems to work fine though. Looking forward to BackupPC 3.0. Thanks for the great product! Nils Breunese. PGP.sig Description: Dit deel van het bericht is digitaal ondertekend - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] BackupPC 2.1.2-5 Reporting XferErrs on Successful Local Restore
I installed BackupPC 2.1.2-5 on an Ubuntu 6.10 server system for local
backups. The only tailoring I needed to do was:
* defined the directories to be backed up
* modified $Conf{TarClientCmd} = '/usr/bin/sudo $tarPath -c -v -f - -C
$shareName+'
. ' --totals';
* added $Conf{TarClientRestoreCmd} = '/usr/bin/sudo $tarPath -x -v -f - -C
$shareName+'
. ' --totals';
Full and incremental backups appear to be working fine. I can
successfully restore files, except BackupPC reports '#xferErrs=1.
Restore#
Result
Start Date
Dur/mins
#files
MB
#tar errs
#xferErrs
5
success
1/20 14:52
0.0
13
10.6
0
1
Error log contains no indication of any problems.
Contents of file /var/lib/backuppc/pc/localhost/RestoreLOG.5, modified
2007-01-20 14:52:11 (Extracting only Errors)
Running: /usr/bin/sudo /bin/tar -x -v -f - -C /home --totals
Running: /usr/share/backuppc/bin/BackupPC_tarCreate -h localhost -n 9 -s
/home -t -r /user -p /user2/ /user
Xfer PIDs are now 6128,6129
tarCreate: Done: 13 files, 11138482 bytes, 2 dirs, 0 specials, 0 errors
Total bytes read: 11151360 (11MiB, 7.9MiB/s)
So far, I have not found which part of the code thinks there is an error
in the restore.
Although a minor problem, any suggestions for getting ride of this issue
would be appreciated!
Thanks, Norbert-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] trashClean start time seems wrong
Tim writes: I agree the display is a little confusing. One change I will consider is to change the start time to reflect when it last woke up. Then it is no longer technically correct, but less confusing. How about changing the label to wake time: then it *is* still accurate. That's not the right term for the other jobs. Plus, any string changes involve translations into several languages. Craig - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC 2.1.2-5 Reporting XferErrs on Successful Local Restore
Norbert writes:
Contents of file /var/lib/backuppc/pc/localhost/RestoreLOG.5, modified
2007-01-20 14:52:11 (Extracting only Errors)
Running: /usr/bin/sudo /bin/tar -x -v -f - -C /home --totals
Running: /usr/share/backuppc/bin/BackupPC_tarCreate -h localhost -n 9 -s
/home -t -r /user -p /user2/ /user
Xfer PIDs are now 6128,6129
tarCreate: Done: 13 files, 11138482 bytes, 2 dirs, 0 specials, 0 errors
Total bytes read: 11151360 (11MiB, 7.9MiB/s)
I suspect it is the last line. In lib/BackupPC/Xfer/Tar.pm, try
changing:
if ( /^Total bytes written: / ) {
to
if ( /^Total bytes (written|read): / ) {
Craig
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC 2.1.2-5 Reporting XferErrs on Successful Local Restore
Craig writes:
I suspect it is the last line. In lib/BackupPC/Xfer/Tar.pm, try
changing:
if ( /^Total bytes written: / ) {
to
if ( /^Total bytes (written|read): / ) {
I should mention this is another change in tar 1-16. Tar 1-15 doesn't
print the string Total bytes read.
Craig
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] tar error 256
Holger writes:
Another possibility could be to write a wrapper around either ssh on the
server or tar on the client to change an exit code of 1 to an exit code of 0,
but that probably has the problem of affecting more serious errors as well
(if it was as simple as patching exit code 1 to 0, I guess there would be a
fix in place already). You could even do this in BackupPC itself, *possibly*
as simple as changing line 213 (in 3.0.0beta3) in Xfer::Tar.pm as in
-if ( !close($t-{pipeTar}) ) {
+if ( !close($t-{pipeTar}) and $? != 256 ) {
but that
a) is *totally* untested,
b) will affect all clients and not only one and
c) will make all failures returning exit code 1 to be regarded as ok
(provided it even works)
- four good reasons not to try it unless you are really desperate :-). With
a wrapper around ssh or tar you can at least limit the effect to one client.
But downgrading tar still seems safest to me.
I hope someone can give you a better solution.
Your solution is fine. I looked at the source for tar 1.16 and it
returns an exit status of 1 only in the case when a file changed
during reading. More specifically:
- the file length changed during reading (eg: more/less bytes
read than the file size returned by stat), or
- the mtime changed while the file was being archived.
d) will of course void your BackupPC warranty ;-)
You are right of course :)
I'll make the change in 2.x and 3.x.
Craig
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
[BackupPC-users] Parent read EOF
I continue to receive child exited prematurely errors when attempting to do an incremental backup of a WinXP client using rsyncd. I have the latest versions of cgywin, rsync, File::RsyncP. Here is an excerpt of the log. I wonder if it has something to do with the double forward slash on the last write before the error. If so, what is the cause and how can it be corrected. Log output: [ skipped 27 lines ] attribWrite(dir=fdocs/Download) - /nasl/backuppc/data/pc/jk8xy01/new/fdocs/fDownload/attrib [ skipped 247 lines ] attribWrite(dir=fdocs) - /nasl/backuppc/data/pc/jk8xy01/new/fdocs/attrib attribWrite(dir=) - /nasl/backuppc/data/pc/jk8xy01/new//attrib Child is aborting Got exit from child Parent read EOF from child: fatal error! Sending csums, cnt = 9359, phase = 1 - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC 2.1.2-5 Reporting XferErrs on Successful Local Restore
Craig, fix to Tar.pm worked like a charm! Thanks, Norbert PS. Great application! Does everything I want it to do, with very little effort on my part. I successfully tested out archiving today as a means of creating monthly offline backups. Next step is backing up Windows workstations. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Avoiding long backup times
On 20/01/07, Holger Parplies [EMAIL PROTECTED] wrote: I read, that backuppc stores every identical file only one time. Basically true, but BackupPC needs to determine that the file is identical to something and to what. Sparing you the transfer (if possible with reasonable cost) is rsyncs job. Doing so comes at a cost in terms of CPU usage, so you've got the option of using tar if bandwidth is cheaper than CPU power. In your case, it obviously is not. [...] What happens with an unchanged file in a full backup? tar and smb will transfer it, rsync will not (presuming you mean unchanged as in same name, same content). For a changed file, rsync will try to speed up the transfer. If you append a few bytes to a large file, tar/smb will transfer the whole file (even on an incremental), while rsync will (basically) transfer some checksums and the few bytes only (on full and incremental backups). So... With tar and smb the file is transfered then it is checked to see if it is the same as a file already in the pool. If it is already in the pool a link is created and the copied file is deleted. Whereas rsync checks if the file is the same as another in the pool before transferring? Have I understood correctly? - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Avoiding long backup times
I asked this at the tail end of an email back in December but no one replied: It seems to me that under most conditions, something with minimal overhead (such as tar) is best for fulls while rsync is best for incrementals. As far as I know, there's no way in backuppc to do this on the same host. Right? If not, is this an idea worth investigating, Craig? Cheers, Stephen -- Stephen Joyce Systems AdministratorP A N I C Physics Astronomy Department Physics Astronomy University of North Carolina at Chapel Hill Network Infrastructure voice: (919) 962-7214and Computing fax: (919) 962-0480 http://www.panic.unc.edu If you want to master emacs, it helps to believe in reincarnation, because there is no way in hell you are going to learn it all in a single lifetime. On Sat, 20 Jan 2007, Nils Breunese (Lemonbit) wrote: Hello all, I think this post by Holger is a pretty good explanation of the differences between the backup types and transfer methods available in BackupPC. Maybe this information could be reworked for inclusion in the BackupPC documentation? Nils Breunese. Holger Parplies wrote: Hi, Clemens von Musil wrote on 20.01.2007 at 00:31:12 [[BackupPC-users] Avoiding long backup times]: [...] One configured backup host has a very slow network connection - a full backup lasts about two days. Because of the exclusive run of backuppc_nightly in 2.xx, I learned, this full backup stalls all other backups. I want to avoid tihs situation and got stuck in following: What exactly happens during full backup? that depends on the transfer method. For a slow connection, you want rsync (or rsyncd), not tar or smb. For a very slow connection, you *definitely* want rsync/rsyncd (which I'll just call rsync for simplicity). I read, that backuppc stores every identical file only one time. Basically true, but BackupPC needs to determine that the file is identical to something and to what. Sparing you the transfer (if possible with reasonable cost) is rsyncs job. Doing so comes at a cost in terms of CPU usage, so you've got the option of using tar if bandwidth is cheaper than CPU power. In your case, it obviously is not. [...] What happens with an unchanged file in a full backup? tar and smb will transfer it, rsync will not (presuming you mean unchanged as in same name, same content). For a changed file, rsync will try to speed up the transfer. If you append a few bytes to a large file, tar/smb will transfer the whole file (even on an incremental), while rsync will (basically) transfer some checksums and the few bytes only (on full and incremental backups). If the file will not be transferred again - what is the difference between full and incremenal? This only applies to rsync (as tar/smb will transfer it). rsync will always transfer missing files as well as update files that have apparently changed. The difference between full and incremental backups lies firstly in the amount of trouble rsync will go to to determine whether a file has changed or not. For an incremental backup, I believe rsync will only look at size and modification time, whereas for a full backup, checksums of the files are calculated, thus consuming much more CPU-time and disk-I/O-bandwidth at least on the client (I understand the server caches checksums if you tell it to), with the benefit of detecting files that have been changed and the modification time reset. The second difference is that a full backup gives a new reference point for following backups while an incremental backup does not necessarily. In version 2 (of BackupPC) all incrementals were relative to the last full (so after one year of only incrementals you'd be transfering everything changed in that year on each incremental backup), whereas version 3 supports multi-level incrementals, with each incremental transfering everything changed since the last incremental of lower level (resp. full backup at level 0). And if yes - does it make sense to keep one full for ever and dealing only with following incrementals? No. With a very slow network connection, you want to avoid transfering changes more often than necessary. Configuring multi-level incrementals in BackupPC 3 seems to be simple enough that you *could* say do incrementals of increasing level each day for the next, say, 10 years, but that will make browsing, restoring and even doing the backups increasingly expensive (and you'll need to keep all the incrementals), even if you *can* neglect modified files your backups are missing. A full backup each day *might* be your best choice, because it avoids duplicating transfers. With any incrementals in between, you'll have at least the fulls re-transfering everything since the last full (less or equal to the sum of the
Re: [BackupPC-users] Avoiding long backup times
Krsnendu dasa wrote: So... With tar and smb the file is transfered then it is checked to see if it is the same as a file already in the pool. If it is already in the pool a link is created and the copied file is deleted. Whereas rsync checks if the file is the same as another in the pool before transferring? Have I understood correctly? Rsync doesn't check against all pooled files, it checks against the same filename in the last backup run - which will be a link to the pooled file. Then on incremental runs it will assume the contents are identical and not transfer anything if the file size and timestamps match. On full runs and size/timestamp mismatches on incrementals, it will then do a block-checksum comparison of the matching files using the rsync algorithm which takes some time but not a lot of bandwidth. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] Pooling Incrementals -- I'm confused
I'm using BackupPC 3.0.0beta3, if it matters. What is the difference between a Full and Incremental backup when Pooling is involved? In a non-pooling backup scheme, a full backup backs /everything/ up, whether or not it's changed since the last full backup. This way, even if File A has not changed, it's still backed up, and if a restore is needed and the current full backup is corrupt, you can always go to the previous one. With BackupPC, as I understand it, there will be only one copy of the file regardless of how many full backups have occurred, and if that one file is corrupt, you're kind of up the creek. Now, it's true that if the file is corrupt it will likely be caught by BackupPC and not be linked to, because it's not identical. But still, there is a time window where this could occur. If my understanding of this is correct, what is the functional difference between a Full and Incremental backup? It's more curiosity, really; I love BackupPC and think it's great! -- Regards, joe Joe Casadonte [EMAIL PROTECTED] -- Llama Fresh Farms = http://www.northbound-train.com Ramblings of a Gay Man = http://www.northbound-train.com/ramblings Emacs Stuff = http://www.northbound-train.com/emacs.html Music CD Trading = http://www.northbound-train.com/cdr.html -- Live Free, that's the message! -- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] Converting from tar to rsync
You know, I am beginning to like the rsync approach better and better. I should be getting some DDR-266 or 400 RAM very soon and will probably opt to make the switch. Is there a way to gracefully switch over from tar to rsync? Or should I just write off the backups I have and start over again? Thanks, --b - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
