On Mon, Sep 17, 2012 at 11:54 AM, Timothy J Massey <tmas...@obscorp.com> wrote:
>
> However, I have recently inherited a server that is >3TB big, and 97%
> full, too! Backups of that system take 3.5 *days* to complete. I *can't*
> live with that. I need better performance.
The only quick-fix would be if there are some top-level directories
where it would make sense to divide the target into several separate
runs. (Different 'hosts' in backuppc, with ClientAliasName pointing
back to the same server and different 'shares' for each). Then you
can skew the schedule so the fulls happen on different days. Or,
schedule this to start on Friday evening and hope it is done by Monday
morning... In any case you have to keep in mind that if you ever
have to restore, there will be considerable downtime. For the price
of a disk these days, it might be worth keeping a copy up to date with
native rsync that you could just swap into place if you needed it and
back that up with backuppc if you want to keep a history.
>
> No matter the size of the system, I seem to top out at about 50GB/hour for
> full backups. Here is a perfectly typical example:
In most systems, the bottleneck is going to be seek time on the
archive disk. That's hard to measure because most linux metrics count
device wait time as CPU time, and it will vary wildly with the size
of the target files. Also, except for the first full it is hard to
relate the xfer speed to the size of the data since you may be reading
a TB to xfer just a MB.
> It looks like everything is under-utilized. For example, I'm getting a
> measly 40-50MB of read performance from my array of four drives,
If every read has to seek many tracks (a likely scenario), that's not
unreasonable performance.
> and
> *nothing* is going out over the network.
If everything you read matches, there won't be much network traffic.
My physical drive and network
> lights echo this: they are *not* busy. My interrupts are certainly
> manageable and context switches are very low. Even my CPU numbers look
> tremendous: nearly no time in wait, and about 50% CPU idle!
I think you should be seeing wait time. Unless perhaps you have some
huge files that end up contiguous on the disk, I'd expect the CPU to
be able to decompress and checksum as fast as the disk can deliver -
and there shouldn't be much other computation involved.
> And one more request: for those of you out there using rsync, can you
> give me some examples where you are getting faster numbers? Let's say, full
> backups of 100GB hosts in roughly 30-35 minutes, or 500GB hosts in two or
> three hours? That's about four times faster than what I'm seeing, and would
> work out to be 50-60MB/s, which seems like a much more realistic speed. If
> you are seeing such speed, can you give us an idea of your hardware
> configuration, as well as an idea of the CPU utilization you're seeing
> during the backups? Also, are you using compression or checksum caching?
> If you need help collecting this info, I'd be happy to help you.
Mine seem to track the target host disk speed more than anything else.
The best I see is 208GB with a full time of 148 minutes. But that
is with backuppc running as a VM on the East coast backing up a target
in California and no particular tuning for efficiency. Compression is
on and no checksum caching.
--
Les Mikesell
lesmikes...@gmail.com
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
