[BackupPC-users] rsync vs rsyncd speed for huge number of small files

2020-04-21 Thread R.C.
Hi What is the expected difference in performance between rsync+shh and rsyncd? I would use it over a private LAN, so no concerns about security. Currently rsync+ssh is way too slow for a huge number of very small files (about 700K email files in an imap server tree), even without --checksum.

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

2020-04-21 Thread Pim Rupert
On 21 Apr 2020, at 12:09, Pim Rupert wrote: > > I am just speculating here, but could it be that when the BackupPC software > is handling the SSH process, it is somehow unable to deal with the implictly > added ProxyCommand arguments? Following this hunch I tried starting the BackupPC_dump

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

2020-04-21 Thread Falko Trojahn via BackupPC-users
Hi Pim, using jumphost here for backing up a remote host and it's VMs without any problems. What BackupPC version do you use? I am using BackupPC 4.3.1-3 from the yum repository for CentOS 7. Very good to hear that you got it working on your installation. ok, so I'll try it on an 4.3.2

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

2020-04-21 Thread Pim Rupert
Hi Falko, > On 21 Apr 2020, at 00:03, Falko Trojahn via BackupPC-users > wrote: > > using jumphost here for backing up a remote host and it's VMs without any > problems. What BackupPC version do you use? I am using BackupPC 4.3.1-3 from the yum repository for CentOS 7. Very good to hear

Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files

2020-04-21 Thread Craig Barratt via BackupPC-users
What version of BackupPC are you running? 4.x will likely be a good deal faster than 3.x for both rsync+ssh and rsyncd. The penalty of rsync+ssh vs rsyncd is likely modest, although it depends on how much data is changing between backups. Craig On Tue, Apr 21, 2020 at 1:33 AM R.C. wrote: >

Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files

2020-04-21 Thread ED Fochler
I would expect no difference for small file performance between rsync and ssh-rsync. The ssh overhead on a modern system limits data rate to something like 75MB/s, nearly saturating a gigabit link. It seems you have basic filesystem performance issues. More RAM, larger caches, SSD?

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

2020-04-21 Thread Pim Rupert
Hi Greg and Falko > On 21 Apr 2020, at 20:09, Greg Harris wrote: > > You seem to have a pretty good handle on stuff, but you aren’t suffering from > CentOS’ silent killer are you? SELinux? Thank for the suggestion. Unfortunately it's not SELinux, I have it fully disabled to be certain. >

Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files

2020-04-21 Thread Richard Shaw
On Tue, Apr 21, 2020 at 11:15 AM ED Fochler wrote: > I would expect no difference for small file performance between rsync and > ssh-rsync. The ssh overhead on a modern system limits data rate to > something like 75MB/s, nearly saturating a gigabit link. It seems you have > basic filesystem

Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files

2020-04-21 Thread R.C.
Il 21/04/2020 18:05, Craig Barratt via BackupPC-users ha scritto: What version of BackupPC are you running?  4.x will likely be a good deal faster than 3.x for both rsync+ssh and rsyncd. I have 3.2.1 backing up this imap folder. I'm evaluating whether to upgrade to V4 but I am mirroring

Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files

2020-04-21 Thread R.C.
Il 21/04/2020 17:48, ED Fochler ha scritto: I would expect no difference for small file performance between rsync and ssh-rsync. The ssh overhead on a modern system limits data rate to something like 75MB/s, nearly saturating a gigabit link. It seems you have basic filesystem performance

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

2020-04-21 Thread Falko Trojahn via BackupPC-users
Hello Pim, using jumphost here for backing up a remote host and it's VMs without any problems. What BackupPC version do you use? I am using BackupPC 4.3.1-3 from the yum repository for CentOS 7. Very good to hear that you got it working on your installation. ok, so I'll try it on an 4.3.2

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

2020-04-21 Thread Greg Harris
You seem to have a pretty good handle on stuff, but you aren’t suffering from CentOS’ silent killer are you? SELinux? Thanks, Greg Harris On Apr 21, 2020, at 8:10 AM, Falko Trojahn via BackupPC-users mailto:backuppc-users@lists.sourceforge.net>> wrote: Hi Pim, using jumphost here for