Re: [BackupPC-users] Enhancing WAN link transfers
David Rees wrote:
> On Thu, Feb 21, 2008 at 11:43 AM, Nick Webb <[EMAIL PROTECTED]> wrote:
>> Rich Rauenzahn wrote:
>
> As Rich said, BackupPC's rsync modules don't support compression. SSH
> compression should work fine, though.
>
> -Dave
>
Yeah, but ssh compression isn't working for me either. Here is my config:
$Conf{RsyncClientCmd} = '$sshPath -C -o CompressionLevel=3 -c
blowfish-cbc -q -x -l root $host $rsyncPath $argList+';
See anything wrong with that? Could be the blowfish crypt option,
that's new as well. It's weird as it works fine for hours, then dies...
same behavior with rsync compression on, which we've concluded shouldn't
work.
Both sides (backuppc server and client) are Ubuntu LTS (6.06) with
backuppc 2.x
Nick
--
Nick Webb
System Administrator
Freelock Computing - www.freelock.com
206.577.0540 x22
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
Nick Webb wrote:
> Rich Rauenzahn wrote:
>> dan wrote:
>>> no, incrementals are more efficient on bandwidth. they do a less
>>> strenuous test to determine if a file has changed.
>>>
>>> at the expense of CPU power on both sides, you can compress the rsync
>>> traffic either with rsync -z
>> Have you tried rsync -z? Last I heard, BackupPC's rsync modules don't
>> support it.
>>
>> Rich
>
> Actually, if I use the -z/--compress option in rsync or use ssh
> compression BackupPC dies after a few hours (aborted by signal=PIPE).
> Any suggestions on how to figure out why this is failing. Works fine
> without compression, but takes forever...
>
> 2008-02-19 14:59:05 full backup started for directory /
> 2008-02-19 22:25:42 Aborting backup up after signal PIPE
> 2008-02-19 22:25:48 Got fatal error during xfer (aborted by signal=PIPE)
> 2008-02-19 22:25:48 Saved partial dump 43
> 2008-02-19 23:00:01 removing incr backup 27
Backuppc's perl version of rsync doesn't support the -z option.
However, it should work to add -C to ssh like this:
$Conf{RsyncClientCmd} = '$sshPath -C -l root $host $rsyncPath $argList+';
--
Les Mikesell
[EMAIL PROTECTED]
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
On Thu, Feb 21, 2008 at 11:43 AM, Nick Webb <[EMAIL PROTECTED]> wrote: > Rich Rauenzahn wrote: > > dan wrote: > >> no, incrementals are more efficient on bandwidth. they do a less > >> strenuous test to determine if a file has changed. > >> > >> at the expense of CPU power on both sides, you can compress the rsync > >> traffic either with rsync -z > > > > Have you tried rsync -z? Last I heard, BackupPC's rsync modules don't > > support it. > > Actually, if I use the -z/--compress option in rsync or use ssh > compression BackupPC dies after a few hours (aborted by signal=PIPE). > Any suggestions on how to figure out why this is failing. Works fine > without compression, but takes forever... As Rich said, BackupPC's rsync modules don't support compression. SSH compression should work fine, though. -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
Rich Rauenzahn wrote: > > dan wrote: >> no, incrementals are more efficient on bandwidth. they do a less >> strenuous test to determine if a file has changed. >> >> at the expense of CPU power on both sides, you can compress the rsync >> traffic either with rsync -z > Have you tried rsync -z? Last I heard, BackupPC's rsync modules don't > support it. > > Rich Actually, if I use the -z/--compress option in rsync or use ssh compression BackupPC dies after a few hours (aborted by signal=PIPE). Any suggestions on how to figure out why this is failing. Works fine without compression, but takes forever... 2008-02-19 14:59:05 full backup started for directory / 2008-02-19 22:25:42 Aborting backup up after signal PIPE 2008-02-19 22:25:48 Got fatal error during xfer (aborted by signal=PIPE) 2008-02-19 22:25:48 Saved partial dump 43 2008-02-19 23:00:01 removing incr backup 27 Nick - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
Rich Rauenzahn wrote: > > dan wrote: >> no, incrementals are more efficient on bandwidth. they do a less >> strenuous test to determine if a file has changed. >> >> at the expense of CPU power on both sides, you can compress the rsync >> traffic either with rsync -z > Have you tried rsync -z? Last I heard, BackupPC's rsync modules don't > support it. > > Rich I tried it a long time back, it was not pretty. Backups failed to complete... after backing up for 24 hours or more. Don't do it. Nick - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
dan wrote: > no, incrementals are more efficient on bandwidth. they do a less > strenuous test to determine if a file has changed. > > at the expense of CPU power on both sides, you can compress the rsync > traffic either with rsync -z Have you tried rsync -z? Last I heard, BackupPC's rsync modules don't support it. Rich - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
i looked at my archive history hear and i have a number of hosts than do
incrementals take like 6 minutes and fulls like 46 minutes
On Feb 19, 2008 4:07 PM, Carl Wilhelm Soderstrom <[EMAIL PROTECTED]>
wrote:
> On 02/19 05:53 , Raman Gupta wrote:
> > Carl Wilhelm Soderstrom wrote:
> > >>> If you're running >=v3 the following option will make all the
> incrementals
> > >>> sync against the previous incremental, instead of the last full.
> This keeps
> > >>> them from growing quite as quickly. (It's the behavior you expect
> from
> > >>> rsync).
> > >>>
> > >>> $Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
> > >>>
> > >> I was under the assumption that BackupPC never transfers the same
> file
> > >> twice unless it changed after the last backup (either full or
> > >> incremental), even in the 2.x version. Was that an invalid
> assumption
> > >> on my part?
> > >
> > > That is incorrect. Backuppc does its incrementals against the last
> full; not
> > > against the previous incremental (unless you set $Conf{IncrLevels} =
> [1, 2,
> > > 3, 4, 5, 6];).
> >
> > So is it correct to say that when using rsync, its probably more
> > efficient to just turn off incrementals and always do fulls?
>
> No, and a brief examination of the reports will make this clear.
> I'm not thoroughly clear on the difference; but backuppc 'fulls' using
> rsync
> do a more thorough set of checks than the 'incrementals'.
>
> Incrementals, even using rsync, are usually much faster than fulls.
>
> --
> Carl Soderstrom
> Systems Administrator
> Real-Time Enterprises
> www.real-time.com
>
> -
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
no, incrementals are more efficient on bandwidth. they do a less strenuous
test to determine if a file has changed.
at the expense of CPU power on both sides, you can compress the rsync
traffic either with rsync -z or if you are using ssh then with ssh's
compression. if you REALLY wanted to go all out you can have rsync pipe
through bzip2 max compression but you would have to pipe it out of bzip2 on
the server side. that would eat up tons of CPU but would likely use less
bandwidth as long as the files you are backing up are not already
compressed.
On Feb 19, 2008 3:53 PM, Raman Gupta <[EMAIL PROTECTED]> wrote:
> Carl Wilhelm Soderstrom wrote:
> >>> If you're running >=v3 the following option will make all the
> incrementals
> >>> sync against the previous incremental, instead of the last full. This
> keeps
> >>> them from growing quite as quickly. (It's the behavior you expect from
> >>> rsync).
> >>>
> >>> $Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
> >>>
> >> I was under the assumption that BackupPC never transfers the same file
> >> twice unless it changed after the last backup (either full or
> >> incremental), even in the 2.x version. Was that an invalid assumption
> >> on my part?
> >
> > That is incorrect. Backuppc does its incrementals against the last full;
> not
> > against the previous incremental (unless you set $Conf{IncrLevels} = [1,
> 2,
> > 3, 4, 5, 6];).
>
> So is it correct to say that when using rsync, its probably more
> efficient to just turn off incrementals and always do fulls?
>
> Cheers,
> Raman Gupta
>
>
> -
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
On 02/19 05:53 , Raman Gupta wrote:
> Carl Wilhelm Soderstrom wrote:
> >>> If you're running >=v3 the following option will make all the incrementals
> >>> sync against the previous incremental, instead of the last full. This
> >>> keeps
> >>> them from growing quite as quickly. (It's the behavior you expect from
> >>> rsync).
> >>>
> >>> $Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
> >>>
> >> I was under the assumption that BackupPC never transfers the same file
> >> twice unless it changed after the last backup (either full or
> >> incremental), even in the 2.x version. Was that an invalid assumption
> >> on my part?
> >
> > That is incorrect. Backuppc does its incrementals against the last full; not
> > against the previous incremental (unless you set $Conf{IncrLevels} = [1, 2,
> > 3, 4, 5, 6];).
>
> So is it correct to say that when using rsync, its probably more
> efficient to just turn off incrementals and always do fulls?
No, and a brief examination of the reports will make this clear.
I'm not thoroughly clear on the difference; but backuppc 'fulls' using rsync
do a more thorough set of checks than the 'incrementals'.
Incrementals, even using rsync, are usually much faster than fulls.
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
Carl Wilhelm Soderstrom wrote:
>>> If you're running >=v3 the following option will make all the incrementals
>>> sync against the previous incremental, instead of the last full. This keeps
>>> them from growing quite as quickly. (It's the behavior you expect from
>>> rsync).
>>>
>>> $Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
>>>
>> I was under the assumption that BackupPC never transfers the same file
>> twice unless it changed after the last backup (either full or
>> incremental), even in the 2.x version. Was that an invalid assumption
>> on my part?
>
> That is incorrect. Backuppc does its incrementals against the last full; not
> against the previous incremental (unless you set $Conf{IncrLevels} = [1, 2,
> 3, 4, 5, 6];).
So is it correct to say that when using rsync, its probably more
efficient to just turn off incrementals and always do fulls?
Cheers,
Raman Gupta
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
> > If you're running >=v3 the following option will make all the incrementals
> > sync against the previous incremental, instead of the last full. This keeps
> > them from growing quite as quickly. (It's the behavior you expect from
> > rsync).
> >
> > $Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
> >
> I was under the assumption that BackupPC never transfers the same file
> twice unless it changed after the last backup (either full or
> incremental), even in the 2.x version. Was that an invalid assumption
> on my part?
That is incorrect. Backuppc does its incrementals against the last full; not
against the previous incremental (unless you set $Conf{IncrLevels} = [1, 2,
3, 4, 5, 6];).
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
Carl Wilhelm Soderstrom wrote:
> On 11/28 09:39 , Tim Hall wrote:
>> Are there any known backuppc tweaks/settings that
>> are proven to increase transfer performance over
>> wan links? Specifically with using rsyncd or rsync
>> as the transfer method.
>
> . . . .
>
> If you're running >=v3 the following option will make all the incrementals
> sync against the previous incremental, instead of the last full. This keeps
> them from growing quite as quickly. (It's the behavior you expect from
> rsync).
>
> $Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
>
>
My apologizes for replying to an ancient thread, but I'm curious if an
upgrade from 2.x to 3.x will help with bandwidth efficiency much?
I was under the assumption that BackupPC never transfers the same file
twice unless it changed after the last backup (either full or
incremental), even in the 2.x version. Was that an invalid assumption
on my part?
Thanks for a great product!
Nick
--
Nick Webb
System Administrator
Freelock Computing - www.freelock.com
206.577.0540 x22
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
On 11/28 09:39 , Tim Hall wrote:
> Are there any known backuppc tweaks/settings that
> are proven to increase transfer performance over
> wan links? Specifically with using rsyncd or rsync
> as the transfer method.
the -C option to compress your SSH data is highly recommended. Also, going
with '-c blowfish-cbc' or '-c arcfour' may reduce CPU usage and increase
transfer rates. You can put this line in config.pl or in a per-host config
file:
$Conf{RsyncClientCmd} = '$sshPath -C -o CompressionLevel=9 -c blowfish-cbc
-q -x -l rsyncbakup $host $rsyncPath $argList+';
the '-l rsyncbakup' is part of my own setup for logging into a remote host
without being root (and from thence using a sudo command in the ssh DSA key
to get root access to files; preventing an attacker from subverting the ssh
command to do arbitrary things to the remote system).
If you're running >=v3 the following option will make all the incrementals
sync against the previous incremental, instead of the last full. This keeps
them from growing quite as quickly. (It's the behavior you expect from
rsync).
$Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
-
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell. From the desktop to the data center, Linux is going
mainstream. Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Enhancing WAN link transfers
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tim Hall wrote: > Hi, > > new to backuppc, I want to use to send my backups > across the Internet (DSL) to an offsite server. > > Are there any known backuppc tweaks/settings that > are proven to increase transfer performance over > wan links? Specifically with using rsyncd or rsync > as the transfer method. > > For example a preferred rsync version. > > I noticed that the transfer(s) seem to run at the max > upstream for the ADSL connection, (72KB), completely > choking it, is this normal? Has anyone else seen this? > Are there any settings to improve this? Using rsync is very helpful (if you enable SSH compression) otherwise you don't get link layer compression. Also, setting the rsync bwlimit can stop you from stealing all available bandwidth, but better to just use QoS to allocate minimum bandwidth to each class, so at night when the link is idle, rsync can use all available bandwidth. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHTlJrGyoxogrTyiURAg+nAKDPR21BE+UpLr/feB54o16S68U+VwCff0ya C5r+cHMaryA03Xcb+pIYTPc= =WDu9 -END PGP SIGNATURE- - SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Enhancing WAN link transfers
Hi, new to backuppc, I want to use to send my backups across the Internet (DSL) to an offsite server. Are there any known backuppc tweaks/settings that are proven to increase transfer performance over wan links? Specifically with using rsyncd or rsync as the transfer method. For example a preferred rsync version. I noticed that the transfer(s) seem to run at the max upstream for the ADSL connection, (72KB), completely choking it, is this normal? Has anyone else seen this? Are there any settings to improve this? Thanks much appreciated. I've noticed that - SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
