Re: [BackupPC-users] Slow local backup
On 06/20 03:57 , Holger Parplies wrote: > Carl W. Soderstrom wrote on 2018-06-14 16:03:24 -0400 [Re: [BackupPC-users] > Slow local backup]: > > On 06/14 03:38 , Bowie Bailey wrote: > > > On 6/14/2018 3:27 PM, Michael Stowe wrote: > > > > Why are you using rsyncd over the loopback instead of ??? rsync? > > > > > > Mainly because that's the way all of my other clients are being backed > > > up [...] > > > > I've always used tar for local backups. The advantage of rsync is greater in > > bandwidth-constrained environments because it saves moving whole files over > > the network. However, if the file needs to be read anyway to see if anything > > has changed, then nothing is saved because the local machine is the same as > > the remote machine. > > well, mostly true. You still save copying large amounts of data from one > process address space to another and possible some context switches. While > that may not make rsync *faster* than tar on local backups, it might mean > it's not much slower. It probably depends on your setup. And it probably > has low enough impact not to worry about it. Thanks for the responses on this, you have some very good knowledge which I've managed to forget over the years. I remember once a long time ago, comparing the speed of an initial backup and finding that tar was 3x-4x faster than rsync for the first data transfer. That showed me how much overhead there was in rsync, and that it may be better to go with tar for at least some backup cases. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
Hi, Carl W. Soderstrom wrote on 2018-06-14 16:03:24 -0400 [Re: [BackupPC-users] Slow local backup]: > On 06/14 03:38 , Bowie Bailey wrote: > > On 6/14/2018 3:27 PM, Michael Stowe wrote: > > > Why are you using rsyncd over the loopback instead of ??? rsync? > > > > Mainly because that's the way all of my other clients are being backed > > up [...] > > I've always used tar for local backups. The advantage of rsync is greater in > bandwidth-constrained environments because it saves moving whole files over > the network. However, if the file needs to be read anyway to see if anything > has changed, then nothing is saved because the local machine is the same as > the remote machine. well, mostly true. You still save copying large amounts of data from one process address space to another and possible some context switches. While that may not make rsync *faster* than tar on local backups, it might mean it's not much slower. It probably depends on your setup. And it probably has low enough impact not to worry about it. > I may be incorrect about some of my understanding here, I know rsync does a > few things which tar does not, but which slip my brain at the moment. tar only has a single timestamp to go by (for incremental backups), rsync has a complete file list. This means that tar will miss *all file deletions* - deleted files will continue to show up in your backups until the next full - as well as renamed or moved files and probably files created with an old timestamp (eg. 'touch --date', 'touch --reference', 'cp -p', 'unzip', 'tar x' etc.). This is important! *tar incrementals will not be exact snapshots*! Well, neither will rsync backups, unless you use a snapshot of the underlying file system (or your file system is quiescent during backup), but that's a different matter. How much effort you should put into getting exact backups depends on what you expect to get out of them. * For restoring the occasional bunch of files from a backup because someone deleted them or messed them up, tar backups are fine. * For restoring a complete file system in the state it was in before it failed, you will want rsync backups, possibly with snapshots (depending on the type of activity on the file system). * If you're not sure what you need, but want to avoid surprises, go with rsync ;-). > Also, some uses of rsync may be more efficient than this by only checking > timestamps. That sounds like rsync incrementals. They will catch * files not in the backup, * files with modified timestamps, and * files with different size, even if the timestamp appears unchanged. Only rsync fulls will actually read all files. My recommendation for local backups: rsync(d). Hope that helps. Regards, Holger -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 2018-06-19 06:55, Bowie Bailey wrote: On 6/19/2018 1:35 AM, Michael Schumacher wrote: Hi there Friday, June 15, 2018, 3:06:30 PM, you wrote: BB> It finally finished after 24 hours. That gives about 13G/hour or about BB> 3.8M/s. BB> The CPUs were not busy. That's what I was confused about. I would have BB> expected to see a bottleneck at some point, but nothing seemed to be BB> busy. The CPUs were all at or below 20% and iowait was close to 0 most BB> of the time. I'm not sure how I would determine if the loopback was BB> saturated. what file system are you using? I am operating a mailserver on a dedicated machine. There are some millions of small files on that machine, so the situation is comparable to yours. When I set up that machine some years ago, I compared various file systems. Surprisingly, ext4 came out as winner to handle these small files. I am just using ext4 on top of the hardware, no lvm on that machine. And of course having a lot of RAM may also speed up your machine, because the server can read ahead data. I think it may be a matter of tuning that server to get better performance. https://u2182357.ct.sendgrid.net/wf/click?upn=rBK8reUlX8Sxr7Iz1fV-2F7QZGjYW6utjZlDnU1Z3QOebvPgcj6rwg-2BwN0ANYujoLploi4MuWYkssMD-2BgaM20V4XxCXAwbqwVkDg6Hcr3r3gg-3D_OypFYCWzG5ApGW-2FFpGTxc4RCS9eud0Dl1htN5rYoUZ8To4zeNUFBkAGI3hzer91CnT16ypaenVxeoDy3w0AsF8dqMUfQGKK-2BfYZDSqtYmGoclGDxESF14nOfj-2Bkqdl2WSEBw4z6PQevIFSOd4lsL3u-2FnZF4LbeZQhQ3zOdQ5LeHhvcRCOai30995dwwynkAYO-2BeXJ5DtQi2U-2FrVdTRMnTworB7cRhz2qnlPweNM9S1xcdyZkUEmauz4qql9a4VA0 may give some hints on read ahead tuning. The backup server is xfs. I haven't looked into performance tuning, since most of my servers are not doing enough IO to need it. It's probably worth mentioning that I experienced a slowness problem on xfs, specifically. Repeated (forced, offline) fscks sometimes turned up subtle filesystem issues, and sometimes reported everything was ok. The symptoms were slowness, and occasional hangs in rsync. My test results: * Moving all files to a new jfs filesystem. Performance improved. No hangs. * Moving all files to a new xfs filesystem. Performance improved. No hangs. * Moving all files to a new ext4 filesystem. Performance improved. No hangs. * Moving all files back to the same physical drives, btrfs. Performance improved. No hangs. My conclusion at the time was that some kind of corruption or decay had crept into the xfs filesystem over time. The physical drives were perfectly fine. I did not diagnose further, but I did abandon xfs and have not had further issues. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/19/2018 1:35 AM, Michael Schumacher wrote: > Hi there > > Friday, June 15, 2018, 3:06:30 PM, you wrote: > > BB> It finally finished after 24 hours. That gives about 13G/hour or about > BB> 3.8M/s. > > BB> The CPUs were not busy. That's what I was confused about. I would have > BB> expected to see a bottleneck at some point, but nothing seemed to be > BB> busy. The CPUs were all at or below 20% and iowait was close to 0 most > BB> of the time. I'm not sure how I would determine if the loopback was > BB> saturated. > > what file system are you using? I am operating a mailserver on a > dedicated machine. There are some millions of small files on that > machine, so the situation is comparable to yours. When I set up that > machine some years ago, I compared various file systems. Surprisingly, > ext4 came out as winner to handle these small files. I am just using > ext4 on top of the hardware, no lvm on that machine. And of course > having a lot of RAM may also speed up your machine, because the server > can read ahead data. I think it may be a matter of tuning that server > to get better performance. > https://www.kernel.org/doc/Documentation/filesystems/ext4.txt may > give some hints on read ahead tuning. The backup server is xfs. I haven't looked into performance tuning, since most of my servers are not doing enough IO to need it. -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
Hi there Friday, June 15, 2018, 3:06:30 PM, you wrote: BB> It finally finished after 24 hours. That gives about 13G/hour or about BB> 3.8M/s. BB> The CPUs were not busy. That's what I was confused about. I would have BB> expected to see a bottleneck at some point, but nothing seemed to be BB> busy. The CPUs were all at or below 20% and iowait was close to 0 most BB> of the time. I'm not sure how I would determine if the loopback was BB> saturated. what file system are you using? I am operating a mailserver on a dedicated machine. There are some millions of small files on that machine, so the situation is comparable to yours. When I set up that machine some years ago, I compared various file systems. Surprisingly, ext4 came out as winner to handle these small files. I am just using ext4 on top of the hardware, no lvm on that machine. And of course having a lot of RAM may also speed up your machine, because the server can read ahead data. I think it may be a matter of tuning that server to get better performance. https://www.kernel.org/doc/Documentation/filesystems/ext4.txt may give some hints on read ahead tuning. best regards --- Michael Schumacher -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/18/2018 10:55 AM, Les Mikesell wrote: > On Mon, Jun 18, 2018 at 9:38 AM, Bowie Bailey wrote: >> I have another big local backup that's been running all weekend. I am >> trying sudo/rsync rather than rsyncd or ssh/rsync this time. This >> directory has millions of small files (and is even bigger than the first >> one), so I expect it to take longer. >> >> This time, I'm seeing higher IO-wait numbers: > Moving the disk head around is almost always the slowest operation and > on the first full it has to traverse every file. And backups tend to > not fit the normal OS caching mechanisms that try to hide that > slowness. Right. I only pointed it out since I did not see this on the previous backup that ran for 24 hours. I'm still not sure where the bottleneck was there since I was never able to see anything that looked busy during the process. -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On Mon, Jun 18, 2018 at 9:38 AM, Bowie Bailey wrote: > > I have another big local backup that's been running all weekend. I am > trying sudo/rsync rather than rsyncd or ssh/rsync this time. This > directory has millions of small files (and is even bigger than the first > one), so I expect it to take longer. > > This time, I'm seeing higher IO-wait numbers: Moving the disk head around is almost always the slowest operation and on the first full it has to traverse every file. And backups tend to not fit the normal OS caching mechanisms that try to hide that slowness. -- Les Mikesell lesmikes...@gmail.com -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/16/2018 9:34 PM, ED Fochler wrote: >> On 2018, Jun 15, at 9:06 AM, Bowie Bailey wrote: >> >> The CPUs were not busy. That's what I was confused about. I would have >> expected to see a bottleneck at some point, but nothing seemed to be >> busy. The CPUs were all at or below 20% and iowait was close to 0 most >> of the time. I'm not sure how I would determine if the loopback was >> saturated. > 20% loaded with 6 or more cores would be a single thread running full speed > on one core. A single task may not stay on one core, it may get moved around > enough that it doesn't appear that any of the cores are heavily loaded. > Doubly true if you have hyperthreading which causes additional CPU shuffling. > > I think you ran into single thread compression performance, plus any of the > normal process, network, and IO latency that would compound with everything > being on one machine. That sounds about right to me. First backups are > slow. SSH probably wouldn't have made any difference at that speed. No, only 2 cores on this machine. I have another big local backup that's been running all weekend. I am trying sudo/rsync rather than rsyncd or ssh/rsync this time. This directory has millions of small files (and is even bigger than the first one), so I expect it to take longer. This time, I'm seeing higher IO-wait numbers: %Cpu0 : 0.0 us, 1.7 sy, 0.0 ni, 5.7 id, 92.6 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 0.3 us, 0.7 sy, 0.0 ni, 86.3 id, 12.4 wa, 0.0 hi, 0.3 si, 0.0 st But I don't see high cpu usage. The IO-wait seems to come and go. It alternates with mostly idle periods: %Cpu0 : 3.0 us, 0.7 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 5.0 us, 2.0 sy, 0.0 ni, 92.3 id, 0.7 wa, 0.0 hi, 0.0 si, 0.0 st Neither cpu seems to go higher than 6%. Once this one finishes, I'll have a first round of full backups for all of the large hosts. Hopefully, the backup times will be a bit more reasonable after that. -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
> On 2018, Jun 15, at 9:06 AM, Bowie Bailey wrote: > > The CPUs were not busy. That's what I was confused about. I would have > expected to see a bottleneck at some point, but nothing seemed to be > busy. The CPUs were all at or below 20% and iowait was close to 0 most > of the time. I'm not sure how I would determine if the loopback was > saturated. 20% loaded with 6 or more cores would be a single thread running full speed on one core. A single task may not stay on one core, it may get moved around enough that it doesn't appear that any of the cores are heavily loaded. Doubly true if you have hyperthreading which causes additional CPU shuffling. I think you ran into single thread compression performance, plus any of the normal process, network, and IO latency that would compound with everything being on one machine. That sounds about right to me. First backups are slow. SSH probably wouldn't have made any difference at that speed. ED. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/14/2018 6:04 PM, Holger Parplies wrote:
> Hi,
>
> Bowie Bailey wrote on 2018-06-14 15:38:44 -0400 [Re: [BackupPC-users] Slow
> local backup]:
>> On 6/14/2018 3:27 PM, Michael Stowe wrote:
>>> On 2018-06-14 10:05, Bowie Bailey wrote:
>>>
>>> I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I
>>> am trying to back up a local 318G filesystem. I am using rsyncd
>>> over the loopback connection. It has been running for 17 hours so
>>> far and has backed up less than half of the directory (based on
>>> the size of the backup filesystem). Running top does not show any
>>> excessive cpu or iowait. ???free??? shows no swap usage and 1.5G
>>> available memory.
> if I'm not miscalculating, that is roughly 10GB/hour or 3MB/s. From what I've
> read about BackupPCv3 performance, that wouldn't seem extremely unreasonable,
> especially for a first backup. For V4, I have no idea what the common figures
> are. The C implementation (rsync_bpc) might have a performance benefit. But I
> would expect one core of your CPU to be almost 100% busy, while the others may
> be idle. Depending on which figure you are looking at, that might not seem
> excessive, but it's the most you can get for single threaded compression
> performance (you *are* using compression, right?).
It finally finished after 24 hours. That gives about 13G/hour or about
3.8M/s.
The CPUs were not busy. That's what I was confused about. I would have
expected to see a bottleneck at some point, but nothing seemed to be
busy. The CPUs were all at or below 20% and iowait was close to 0 most
of the time. I'm not sure how I would determine if the loopback was
saturated.
>
>> [...]
>> I use rsyncd rather than rsync to avoid the ssh overhead.
> For a local backup, you can achieve the same by using 'sudo' instead of 'ssh'.
> At least you could with V3. I've forgotten if and how you can use 'sudo' with
> rsync in V4.
>
>> I expected a backup done via the loopback interface to be fast since it
>> doesn't have the normal networking bandwidth limitations.
> Well, yes, but you still have a network stack and probably at least two
> copies
> between kernel and user space. loopback networking does not come for free. A
> quick 'netperf' test gives me a bit less than 9 Gbit/s throughput. That's
> certainly faster than Gbit networking, but only by a factor of 10.
My network is 100M, so that is significantly faster than what I'm used
to. Also, *only* 10 times faster?? :)
> More importantly, the other limitations don't change - disk and compression
> speed, for instance. If your bottleneck is not the network, a faster network
> won't change anything. Keep in mind that a local backup means that client
> and server are the same computer, so a loopback backup might actually be
> *slower* than a remote backup. Your source file system and BackupPC pool are
> on different physical disks, hopefully?
Yes, the OS and BackupPC pool are on one pair of mirrored SATA disks and
the data that I'm backing up is on another pair of disks.
>
>>> Is it normal for the backup to take this long?
>>>
>>> While that's hard to guess without knowing the particulars of your
>>> system, I'm going to go out on a limb and say, no. No it is not.
> I believe it is still important whether it is the first backup or not. The
> recommendation used to be "ignore the timing of the first backup - fix your
> problem only if the second (or third) backup is still too slow". That would
> still seem to apply to BackupPCv4.
True. I haven't done a first backup in quite a while, so I guess I've
forgotten how slow they can be.
>
>>> Is there a better way to back up a local filesystem?
>>>
>>> Personally, I use rsync (not rsyncd) and in the cases where I have
>>> experienced slowness, it was due to poorly chosen rsync parameters (I
>>> note that this would not differ between rsync and rsyncd), a broken
>>> filesystem, or a specific bug in a specific version of rsync.
>> How do you go about setting up rsync so that it does a local copy rather
>> than going through ssh over the network?
> I would try setting $Conf {RsyncSshArgs} = [ '-e', '/usr/bin/sudo' ], but
> you might run into quoting problems. You could then try using a script
> containing something like 'exec /usr/bin/sudo $@'. Or see if using 'ssh'
> makes much of a difference at all ...
That sounds like it might work. I'll set up a test host to backup a
single directory and play around with it.
I can't see how rsync over ssh could possibly be any faster than a
direct connection to rsyncd.
--
Bowie
-
Re: [BackupPC-users] Slow local backup
On 06/14 03:38 , Bowie Bailey wrote: > On 6/14/2018 3:27 PM, Michael Stowe wrote: > > Why are you using rsyncd over the loopback instead of … rsync? > > > > Mainly because that's the way all of my other clients are being backed > up and what I found when I searched for how to back up a local > filesystem said to do it the same way as the others and just point it to > localhost. I use rsyncd rather than rsync to avoid the ssh overhead. I > expected a backup done via the loopback interface to be fast since it > doesn't have the normal networking bandwidth limitations. I've always used tar for local backups. The advantage of rsync is greater in bandwidth-constrained environments because it saves moving whole files over the network. However, if the file needs to be read anyway to see if anything has changed, then nothing is saved because the local machine is the same as the remote machine. I may be incorrect about some of my understanding here, I know rsync does a few things which tar does not, but which slip my brain at the moment. Also, some uses of rsync may be more efficient than this by only checking timestamps. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
Hi,
Bowie Bailey wrote on 2018-06-14 15:38:44 -0400 [Re: [BackupPC-users] Slow
local backup]:
> On 6/14/2018 3:27 PM, Michael Stowe wrote:
> > On 2018-06-14 10:05, Bowie Bailey wrote:
> >
> > I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I
> > am trying to back up a local 318G filesystem. I am using rsyncd
> > over the loopback connection. It has been running for 17 hours so
> > far and has backed up less than half of the directory (based on
> > the size of the backup filesystem). Running top does not show any
> > excessive cpu or iowait. ???free??? shows no swap usage and 1.5G
> > available memory.
if I'm not miscalculating, that is roughly 10GB/hour or 3MB/s. From what I've
read about BackupPCv3 performance, that wouldn't seem extremely unreasonable,
especially for a first backup. For V4, I have no idea what the common figures
are. The C implementation (rsync_bpc) might have a performance benefit. But I
would expect one core of your CPU to be almost 100% busy, while the others may
be idle. Depending on which figure you are looking at, that might not seem
excessive, but it's the most you can get for single threaded compression
performance (you *are* using compression, right?).
> [...]
> I use rsyncd rather than rsync to avoid the ssh overhead.
For a local backup, you can achieve the same by using 'sudo' instead of 'ssh'.
At least you could with V3. I've forgotten if and how you can use 'sudo' with
rsync in V4.
> I expected a backup done via the loopback interface to be fast since it
> doesn't have the normal networking bandwidth limitations.
Well, yes, but you still have a network stack and probably at least two copies
between kernel and user space. loopback networking does not come for free. A
quick 'netperf' test gives me a bit less than 9 Gbit/s throughput. That's
certainly faster than Gbit networking, but only by a factor of 10.
More importantly, the other limitations don't change - disk and compression
speed, for instance. If your bottleneck is not the network, a faster network
won't change anything. Keep in mind that a local backup means that client
and server are the same computer, so a loopback backup might actually be
*slower* than a remote backup. Your source file system and BackupPC pool are
on different physical disks, hopefully?
> > Is it normal for the backup to take this long?
> >
> > While that's hard to guess without knowing the particulars of your
> > system, I'm going to go out on a limb and say, no. No it is not.
I believe it is still important whether it is the first backup or not. The
recommendation used to be "ignore the timing of the first backup - fix your
problem only if the second (or third) backup is still too slow". That would
still seem to apply to BackupPCv4.
> > Is there a better way to back up a local filesystem?
> >
> > Personally, I use rsync (not rsyncd) and in the cases where I have
> > experienced slowness, it was due to poorly chosen rsync parameters (I
> > note that this would not differ between rsync and rsyncd), a broken
> > filesystem, or a specific bug in a specific version of rsync.
>
> How do you go about setting up rsync so that it does a local copy rather
> than going through ssh over the network?
I would try setting $Conf {RsyncSshArgs} = [ '-e', '/usr/bin/sudo' ], but
you might run into quoting problems. You could then try using a script
containing something like 'exec /usr/bin/sudo $@'. Or see if using 'ssh'
makes much of a difference at all ...
Hope that helps.
Regards,
Holger
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
FWIW, here's my localhost.pl file:
#
# Local server backup of /etc as user backuppc
#
$Conf{XferMethod} = 'tar';
#for some reason pings fail
$Conf{PingCmd} = '/bin/true';
$Conf{BackupFilesExclude} = ['/proc', '/sys', '/var/lib/backuppc', '/var/log',
'/tmp', '/var/tmp', '/staff', '/mnt'];
#$Conf{TarShareName} = ['/etc', '/var/lib/backuppc/.ssh/', '/root'];
$Conf{TarShareName} = ['/'];
$Conf{TarClientCmd} = '/usr/bin/env LC_ALL=C /usr/bin/sudo $tarPath -c -v -f -
-C $shareName'
. ' --totals';
# turning off compression on these files, so they can be recovered without
# backuppc.
# wouldn't make sense to need your backup server,
# in order to recover your backup server, now would it?
$Conf{CompressLevel} = 0;
# do backups anytime
$Conf{BlackoutPeriods} = [];
# remove extra shell escapes ($fileList+ etc.) that are
# needed for remote backups but may break local ones
#$Conf{TarFullArgs} = '$fileList';
#$Conf{TarIncrArgs} = '--newer=$incrDate $fileList';
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 6/14/2018 3:27 PM, Michael Stowe wrote: > > On 2018-06-14 10:05, Bowie Bailey wrote: > > I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I > am trying to back up a local 318G filesystem. I am using rsyncd > over the loopback connection. It has been running for 17 hours so > far and has backed up less than half of the directory (based on > the size of the backup filesystem). Running top does not show any > excessive cpu or iowait. “free” shows no swap usage and 1.5G > available memory. > > Why are you using rsyncd over the loopback instead of … rsync? > Mainly because that's the way all of my other clients are being backed up and what I found when I searched for how to back up a local filesystem said to do it the same way as the others and just point it to localhost. I use rsyncd rather than rsync to avoid the ssh overhead. I expected a backup done via the loopback interface to be fast since it doesn't have the normal networking bandwidth limitations. > Is it normal for the backup to take this long? > > While that's hard to guess without knowing the particulars of your > system, I'm going to go out on a limb and say, no. No it is not. > > Is there a better way to back up a local filesystem? > > Personally, I use rsync (not rsyncd) and in the cases where I have > experienced slowness, it was due to poorly chosen rsync parameters (I > note that this would not differ between rsync and rsyncd), a broken > filesystem, or a specific bug in a specific version of rsync. > How do you go about setting up rsync so that it does a local copy rather than going through ssh over the network? -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow local backup
On 2018-06-14 10:05, Bowie Bailey wrote: I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I am trying to back up a local 318G filesystem. I am using rsyncd over the loopback connection. It has been running for 17 hours so far and has backed up less than half of the directory (based on the size of the backup filesystem). Running top does not show any excessive cpu or iowait. "free" shows no swap usage and 1.5G available memory. Why are you using rsyncd over the loopback instead of ... rsync? Is it normal for the backup to take this long? While that's hard to guess without knowing the particulars of your system, I'm going to go out on a limb and say, no. No it is not. Is there a better way to back up a local filesystem? Personally, I use rsync (not rsyncd) and in the cases where I have experienced slowness, it was due to poorly chosen rsync parameters (I note that this would not differ between rsync and rsyncd), a broken filesystem, or a specific bug in a specific version of rsync. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Slow local backup
I just installed BackupPC v4 on a CentOS 7 server with 4G ram. I am trying to back up a local 318G filesystem. I am using rsyncd over the loopback connection. It has been running for 17 hours so far and has backed up less than half of the directory (based on the size of the backup filesystem). Running top does not show any excessive cpu or iowait. "free" shows no swap usage and 1.5G available memory. Is it normal for the backup to take this long? Is there a better way to back up a local filesystem? -- Bowie -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
