[BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Mark Coetser
Hi backuppc 3.1.0-9.1 rsync 3.0.7-2 OK I have a fairly decent spec backup server with 2 gigabit e1000 nics bonned together and running in bond mode 0 all working 100%. If I run plain rsync between the backup server

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Tim Fletcher
You are being hit by disk io speeds, check you dont have atime turned on on the fs. Also it's worth considering tar instead of rsync for this sort of work load. -- Sent from a mobile device Tim Fletcher On 17 Sep 2012, at 10:08, Mark Coetser m...@tux-edo.co.za wrote: Hi backuppc

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Mark Coetser
On 17/09/2012 14:50, Tim Fletcher wrote: You are being hit by disk io speeds, check you dont have atime turned on on the fs. Also it's worth considering tar instead of rsync for this sort of work load. -- Hi Surely disk io would affect normal rsync as well? Normal rsync and even nfs get

Re: [BackupPC-users] Getting files newer than error

2012-09-17 Thread Michael Stowe
On 2012-09-14 13:46, Michael Stowe wrote: The Linux kernel keeps its own time, generally in UTC. This is translated to local time depending on the environment process of the user itself. Thus, the backuppc user can have a completely different time than, say, you, or root. Depending on

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Frédéric Massot
Le 17/09/2012 14:50, Tim Fletcher a écrit : You are being hit by disk io speeds, check you dont have atime turned on on the fs. Also it's worth considering tar instead of rsync for this sort of work load. Hi, Is the relatime option is acceptable to replace atime? Regards. --

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Tyler J. Wagner
On 2012-09-17 14:18, Frédéric Massot wrote: Is the relatime option is acceptable to replace atime? Unless you are using mutt on the BackupPC server, use noatime. There is no longer any common use for file access time. Regards, Tyler -- We should forget about small efficiencies, say about 97%

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Les Mikesell
On Mon, Sep 17, 2012 at 7:59 AM, Mark Coetser m...@tux-edo.co.za wrote: Surely disk io would affect normal rsync as well? Normal rsync and even nfs get normal transfer speeds its only rsync within backuppc that is slow. Backuppc uses its own rsync implementation in perl on the server side so

Re: [BackupPC-users] Getting files newer than error

2012-09-17 Thread Les Mikesell
On Mon, Sep 17, 2012 at 8:38 AM, Michael Stowe mst...@chicago.us.mensa.org wrote: That's absolutely true -- it's this line: Running: /usr/bin/smbclient ts-1\\Users -c tarmode\ full -TcN /var/lib/BackupPC//pc/ts-1/timeStamp.level0 - That leads me to believe that the error is caused by

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Mark Coetser
On 17/09/2012 17:01, Les Mikesell wrote: On Mon, Sep 17, 2012 at 7:59 AM, Mark Coetserm...@tux-edo.co.za wrote: Surely disk io would affect normal rsync as well? Normal rsync and even nfs get normal transfer speeds its only rsync within backuppc that is slow. Backuppc uses its own rsync

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Les Mikesell
On Mon, Sep 17, 2012 at 10:16 AM, Mark Coetser m...@tux-edo.co.za wrote: Its the first full run but its taking forever to complete, it was running for nearly 3 days! As long is it makes it through, don't make any judgements until after the 3nd full, and be sure you have set up checksum

Re: [BackupPC-users] Backup aborted (Tar exited with error 512 () status)

2012-09-17 Thread Dave Williams
Can anyone provide any help on this? I am at a total loss to understand why I am getting this error of what is even causing this. On 9/13/2012 4:01 PM, Dave Williams wrote: All, Started to get this error during my

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Timothy J Massey
Les Mikesell lesmikes...@gmail.com wrote on 09/17/2012 11:51:09 AM: On Mon, Sep 17, 2012 at 10:16 AM, Mark Coetser m...@tux-edo.co.za wrote: Its the first full run but its taking forever to complete, it was running for nearly 3 days! As long is it makes it through, don't make any

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Rodrigo Severo
On Mon, Sep 17, 2012 at 12:16 PM, Mark Coetser m...@tux-edo.co.za wrote: Its the first full run but its taking forever to complete, it was running for nearly 3 days! I'm seeing similar issues here. Is there any troubleshooting recommended to this kind of problem? Rodrigo

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Timothy J Massey
Mark Coetser m...@tux-edo.co.za wrote on 09/17/2012 03:08:49 AM: Hi backuppc 3.1.0-9.1 rsync 3.0.7-2 OK I have a fairly decent spec backup server with 2 gigabit e1000 nics bonned together and running in bond

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Timothy J Massey
Les Mikesell lesmikes...@gmail.com wrote on 09/17/2012 11:01:25 AM: On Mon, Sep 17, 2012 at 7:59 AM, Mark Coetser m...@tux-edo.co.za wrote: Surely disk io would affect normal rsync as well? Normal rsync and even nfs get normal transfer speeds its only rsync within backuppc that is slow.

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Timothy J Massey
Mark Coetser m...@tux-edo.co.za wrote on 09/17/2012 11:16:29 AM: Its the first full run but its taking forever to complete, it was running for nearly 3 days! *IF* the backup is bandwidth-limited, the first run will take longer than subsequent runs. How much depends on how bandwidth-limited

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Timothy J Massey
Rodrigo Severo rodr...@fabricadeideias.com wrote on 09/17/2012 11:22:23 AM: On Mon, Sep 17, 2012 at 12:16 PM, Mark Coetser m...@tux-edo.co.za wrote: Its the first full run but its taking forever to complete, it was running for nearly 3 days! I'm seeing similar issues here. Is there

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Les Mikesell
On Mon, Sep 17, 2012 at 11:05 AM, Timothy J Massey tmas...@obscorp.com wrote: I'm writing a longer reply, but here's a quick in-thread reply: I know exactly what you mean by waiting until after the first full. Often the second full will be faster -- but only *IF* you are bandwidth limited

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread John Rouillard
On Mon, Sep 17, 2012 at 12:58:28PM -0400, Timothy J Massey wrote: Les Mikesell lesmikes...@gmail.com wrote on 09/17/2012 11:01:25 AM: On the 2nd it will read/uncompress everything for block-checksum verification. If you have enabled checksum caching, fulls after the 2nd will not have to

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread John Rouillard
On Mon, Sep 17, 2012 at 12:54:35PM -0400, Timothy J Massey wrote: No matter the size of the system, I seem to top out at about 50GB/hour for full backups. Here is a perfectly typical example: Full Backup: 769.3 minutes for 675677.3MB of data. That works out to be 878MB/min, or about

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread John Rouillard
On Mon, Sep 17, 2012 at 06:05:28PM +, John Rouillard wrote: On Mon, Sep 17, 2012 at 12:54:35PM -0400, Timothy J Massey wrote: No matter the size of the system, I seem to top out at about 50GB/hour for full backups. Here is a perfectly typical example: Full Backup: 769.3 minutes

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Les Mikesell
On Mon, Sep 17, 2012 at 11:54 AM, Timothy J Massey tmas...@obscorp.com wrote: However, I have recently inherited a server that is 3TB big, and 97% full, too! Backups of that system take 3.5 *days* to complete. I *can't* live with that. I need better performance. The only quick-fix would be

Re: [BackupPC-users] backuppc slow rsync speeds

2012-09-17 Thread Tim Fletcher
No it won't in the same way, you are basically asking rsync to walk the large and complex file tree checking the date of every file, where as with a full rsync all you are asking for is next file, next lie, next file -- Sent from a mobile device On 17 Sep 2012, at 15:59, Mark Coetser