Marco Nicolayevsky <ma...@specialtyvalvegroup.com> wrote on 05/14/2014
08:47:24 PM:
> Hello all,
>
> I am using a pretty vanilla installation of BackupPC and love the
> simplicity and fact that it just works as it’s supposed to.
>
> My problem arises when trying to back up windows clients over rsyncd.
>
> I have 3 win boxes, all running Win7 and the rsncd server.
> Functionally, they perform fine. I’m able to backup, restore, etc.
> My initial tests were with small folders and not the entire drive.
>
> HOWEVER… each win box has somewhere between 20GB and 3TB of data,
> and despite being on wired gigabit Ethernet, the 3TB machine ran for
> over 8 days and still wasn’t done before I pulled the plug and
> called it quits. Backing up 3TB over gig-e should be able to be
> accomplished in under 1 day, so I’m at a loss of what to do next.
Here are some hard numbers for one of my systems. All targets
Windows-based, using rsyncd, no SSH, no encryption, no compression.
Servers are using 7200RPM SATA drives.
Full size: 1372GB, 1114587 files. Time to complete: 1175.3 minutes.
(Intel Xeon E5 CPU, 16GB RAM, 6-drive Software RAID-6)
Full size: 627GB, 452759 files. Time to complete: 682.4 minutes.
(Intel Atom D610 (really slow) CPU, 4GB RAM, 5-drive Software RAID-6)
Now, these are established backups. I've found that initial backups can
be 2-3 times as long. And at 3TB, you're nearly triple of the bigger one.
So, that could mean that the initial backup can take 6000 minutes or
more, or four days. And personally, I've found that compression can
double or triple the time again. Worst-case of all of that could be two
weeks! So it is possible that you're seeing correct operation for that
first backup, and that future backups should be much better.
Also, those are fulls. Incrementals take 4 hours and 1 hour,
respectively.
> Is rsync “really” that slow?
No: rsync is only going to be marginally slower than a non-rsync copy,
even on the first time, assuming you're not limited by something else (CPU
or RAM) that would not be a limit for a normal copy.
That could be related to the number of files: that's an area where rsync
can get tripped up. As you can see, I've got >1 million files, so the
definition of "too many" is pretty big. But if you had, say, 10M files,
maybe that's an issue to consider.
> Am I doing something wrong?
*You*? Who knows? Is something wrong? Possibly.
> Is it
> limited to just the windows client?
No.
> When I was evaluating Bacula, I
> was able to do the same backup in 1/10th the time (or at least it
> felt that way since I don’t have hard numbers).
You've got hard numbers above to consider as a comparison and see if they
will fit your needs.
> Before giving up on BackupPC and considering an alternative, can
> someone give me some advice?
Yes. First, you've given us *NOTHING* to go on other than "it's slow".
It's like telling a mechanic "my car doesn't run right." Of course,
you're probably expecting to pay the mechanic, so he's incented to ask
lots of questions to figure things out. I think it's pretty telling that
I can't see a single reply to your request. While your request includes a
lot of words, there was almost *NOTHING* of substance in your request
except a *VERY* brief description of your hardware. We have almost no
description of what your backups look like (size and number of files, for
example), what your backup jobs look like (compression being a very big
one), or what your backup server is doing (CPU, memory or disk usage). And
we're not getting paid, so we're not really incented to ask you a lot of
questions. But I'll give you a few:
First, is your system limited in some way? Are you hitting 100% CPU, or
swapping (not enough RAM), or are your disk overloaded, or something else?
Learn to use top and vmstat (or dstat).
Is your infrastructure fundamentally limited in some way? Have you tried
doing a straight-up copy from a target to your backup system to make sure
that the underlying infrastructure is capable of delivering what you
expect it to? If you can only get 1-2MB/s copies using SMB, tar, NFS,
FTP, etc. , then that's all you'll get with BackupPC, too. But if you can
get 70MB/s copies between the same two systems some other way, then we can
expect better of BackupPC. (But all that does is re-ask the question of
what is limiting you.)
From my e-mail, you know what is possible and reasonable to get. If
you're far away from those results, then you need to figure out what is
different about your system and causing the slowdown.
The second thing to try is to simplify things. For me, the first thing I
do is disable compression. In today's multi-core universe, compression is
rapidly becoming a bottleneck again. The compression algorithms in common
use today do *not* use multiple cores. On a system with more than a
couple of disks I can easily max out one core with compression.
Or, try to use SMB instead of rsyncd. I would not suggest that as a final
stop (I really like rsyncd for my systems), but it will help you to see if
there is a problem with something unique to rsync, rather than
fundamentally wrong with your system.
Another option would be to expand your testing from a very small section
to something larger: say, 100GB. That is big enough to be somewhat
representative of the whole, but should be able to complete quickly enough
even with compression, encryption, etc. to get some baseline numbers to
work with, including both the *first* and *additional* full backups. That
way, you might find that the initial backup will take a week, but each
additional backup after that will only take 12 hours and you're OK with
that. Or, you might find that things are still broken, but now it won't
cost you a week of your life every time you want to test.
> If it helps, the server is beefy, quad-core i5 proc w/ 4gb ram and 6TB
raid5.
Eh. That's pretty baseline for anything you would call a server today,
and an i5 processor screams client-grade, not server grade. It's not
especially beefy, but BackupPC doesn't really require beefy. See my stats
on a really crummy embedded-class CPU and software RAID-6! :)
Tim Massey
Out of the Box Solutions, Inc.
Creative IT Solutions Made Simple!
http://www.OutOfTheBoxSolutions.com
tmas...@obscorp.com
22108 Harper Ave.
St. Clair Shores, MI 48080
Office: (800)750-4OBS (4627)
Cell: (586)945-8796
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
