Re: [BackupPC-users] Slow local backup
On 06/14 03:38 , Bowie Bailey wrote: > On 6/14/2018 3:27 PM, Michael Stowe wrote: > > Why are you using rsyncd over the loopback instead of … rsync? > > > > Mainly because that's the way all of my other clients are being backed > up and what I found when I searched for how to back up a local > filesystem said to do it the same way as the others and just point it to > localhost. I use rsyncd rather than rsync to avoid the ssh overhead. I > expected a backup done via the loopback interface to be fast since it > doesn't have the normal networking bandwidth limitations. I've always used tar for local backups. The advantage of rsync is greater in bandwidth-constrained environments because it saves moving whole files over the network. However, if the file needs to be read anyway to see if anything has changed, then nothing is saved because the local machine is the same as the remote machine. I may be incorrect about some of my understanding here, I know rsync does a few things which tar does not, but which slip my brain at the moment. Also, some uses of rsync may be more efficient than this by only checking timestamps. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
Hi,
Bowie Bailey wrote on 2018-06-14 15:38:44 -0400 [Re: [BackupPC-users] Slow
local backup]:
> On 6/14/2018 3:27 PM, Michael Stowe wrote:
> > On 2018-06-14 10:05, Bowie Bailey wrote:
> >
> > I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I
> > am trying to back up a local 318G filesystem. I am using rsyncd
> > over the loopback connection. It has been running for 17 hours so
> > far and has backed up less than half of the directory (based on
> > the size of the backup filesystem). Running top does not show any
> > excessive cpu or iowait. ???free??? shows no swap usage and 1.5G
> > available memory.
if I'm not miscalculating, that is roughly 10GB/hour or 3MB/s. From what I've
read about BackupPCv3 performance, that wouldn't seem extremely unreasonable,
especially for a first backup. For V4, I have no idea what the common figures
are. The C implementation (rsync_bpc) might have a performance benefit. But I
would expect one core of your CPU to be almost 100% busy, while the others may
be idle. Depending on which figure you are looking at, that might not seem
excessive, but it's the most you can get for single threaded compression
performance (you *are* using compression, right?).
> [...]
> I use rsyncd rather than rsync to avoid the ssh overhead.
For a local backup, you can achieve the same by using 'sudo' instead of 'ssh'.
At least you could with V3. I've forgotten if and how you can use 'sudo' with
rsync in V4.
> I expected a backup done via the loopback interface to be fast since it
> doesn't have the normal networking bandwidth limitations.
Well, yes, but you still have a network stack and probably at least two copies
between kernel and user space. loopback networking does not come for free. A
quick 'netperf' test gives me a bit less than 9 Gbit/s throughput. That's
certainly faster than Gbit networking, but only by a factor of 10.
More importantly, the other limitations don't change - disk and compression
speed, for instance. If your bottleneck is not the network, a faster network
won't change anything. Keep in mind that a local backup means that client
and server are the same computer, so a loopback backup might actually be
*slower* than a remote backup. Your source file system and BackupPC pool are
on different physical disks, hopefully?
> > Is it normal for the backup to take this long?
> >
> > While that's hard to guess without knowing the particulars of your
> > system, I'm going to go out on a limb and say, no. No it is not.
I believe it is still important whether it is the first backup or not. The
recommendation used to be "ignore the timing of the first backup - fix your
problem only if the second (or third) backup is still too slow". That would
still seem to apply to BackupPCv4.
> > Is there a better way to back up a local filesystem?
> >
> > Personally, I use rsync (not rsyncd) and in the cases where I have
> > experienced slowness, it was due to poorly chosen rsync parameters (I
> > note that this would not differ between rsync and rsyncd), a broken
> > filesystem, or a specific bug in a specific version of rsync.
>
> How do you go about setting up rsync so that it does a local copy rather
> than going through ssh over the network?
I would try setting $Conf {RsyncSshArgs} = [ '-e', '/usr/bin/sudo' ], but
you might run into quoting problems. You could then try using a script
containing something like 'exec /usr/bin/sudo $@'. Or see if using 'ssh'
makes much of a difference at all ...
Hope that helps.
Regards,
Holger
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
FWIW, here's my localhost.pl file:
#
# Local server backup of /etc as user backuppc
#
$Conf{XferMethod} = 'tar';
#for some reason pings fail
$Conf{PingCmd} = '/bin/true';
$Conf{BackupFilesExclude} = ['/proc', '/sys', '/var/lib/backuppc', '/var/log',
'/tmp', '/var/tmp', '/staff', '/mnt'];
#$Conf{TarShareName} = ['/etc', '/var/lib/backuppc/.ssh/', '/root'];
$Conf{TarShareName} = ['/'];
$Conf{TarClientCmd} = '/usr/bin/env LC_ALL=C /usr/bin/sudo $tarPath -c -v -f -
-C $shareName'
. ' --totals';
# turning off compression on these files, so they can be recovered without
# backuppc.
# wouldn't make sense to need your backup server,
# in order to recover your backup server, now would it?
$Conf{CompressLevel} = 0;
# do backups anytime
$Conf{BlackoutPeriods} = [];
# remove extra shell escapes ($fileList+ etc.) that are
# needed for remote backups but may break local ones
#$Conf{TarFullArgs} = '$fileList';
#$Conf{TarIncrArgs} = '--newer=$incrDate $fileList';
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/14/2018 3:27 PM, Michael Stowe wrote: > > On 2018-06-14 10:05, Bowie Bailey wrote: > > I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I > am trying to back up a local 318G filesystem. I am using rsyncd > over the loopback connection. It has been running for 17 hours so > far and has backed up less than half of the directory (based on > the size of the backup filesystem). Running top does not show any > excessive cpu or iowait. “free” shows no swap usage and 1.5G > available memory. > > Why are you using rsyncd over the loopback instead of … rsync? > Mainly because that's the way all of my other clients are being backed up and what I found when I searched for how to back up a local filesystem said to do it the same way as the others and just point it to localhost. I use rsyncd rather than rsync to avoid the ssh overhead. I expected a backup done via the loopback interface to be fast since it doesn't have the normal networking bandwidth limitations. > Is it normal for the backup to take this long? > > While that's hard to guess without knowing the particulars of your > system, I'm going to go out on a limb and say, no. No it is not. > > Is there a better way to back up a local filesystem? > > Personally, I use rsync (not rsyncd) and in the cases where I have > experienced slowness, it was due to poorly chosen rsync parameters (I > note that this would not differ between rsync and rsyncd), a broken > filesystem, or a specific bug in a specific version of rsync. > How do you go about setting up rsync so that it does a local copy rather than going through ssh over the network? -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 2018-06-14 10:05, Bowie Bailey wrote: I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I am trying to back up a local 318G filesystem. I am using rsyncd over the loopback connection. It has been running for 17 hours so far and has backed up less than half of the directory (based on the size of the backup filesystem). Running top does not show any excessive cpu or iowait. "free" shows no swap usage and 1.5G available memory. Why are you using rsyncd over the loopback instead of ... rsync? Is it normal for the backup to take this long? While that's hard to guess without knowing the particulars of your system, I'm going to go out on a limb and say, no. No it is not. Is there a better way to back up a local filesystem? Personally, I use rsync (not rsyncd) and in the cases where I have experienced slowness, it was due to poorly chosen rsync parameters (I note that this would not differ between rsync and rsyncd), a broken filesystem, or a specific bug in a specific version of rsync. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Slow local backup
I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I am trying to back up a local 318G filesystem. I am using rsyncd over the loopback connection. It has been running for 17 hours so far and has backed up less than half of the directory (based on the size of the backup filesystem). Running top does not show any excessive cpu or iowait. "free" shows no swap usage and 1.5G available memory. Is it normal for the backup to take this long? Is there a better way to back up a local filesystem? -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
