Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files
On Tue, Apr 21, 2020 at 11:15 AM ED Fochler wrote: > I would expect no difference for small file performance between rsync and > ssh-rsync. The ssh overhead on a modern system limits data rate to > something like 75MB/s, nearly saturating a gigabit link. It seems you have > basic filesystem performance issues. More RAM, larger caches, SSD? > investigate with iostat on client or server? If the host has a CPU bottleneck an easy solution (for the OP's indication it was a private network) would be to change the cipher ssh uses to something less intensive. Add "-c blowfish" to the ssh options, just keep in mind the host system will have to allow that (weak) cipher... Thanks, Richard ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files
Il 21/04/2020 17:48, ED Fochler ha scritto: I would expect no difference for small file performance between rsync and ssh-rsync. The ssh overhead on a modern system limits data rate to something like 75MB/s, nearly saturating a gigabit link. It seems you have basic filesystem performance issues. More RAM, larger caches, SSD? investigate with iostat on client or server? I don't have speed issues on this filesystem during normal use; althought I have not made a specific benchmarking session. Anyhow, it is true that XFS is not the best option for many small files. I'm planning to move to EXT4 in the near future. Thank you Raf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files
Il 21/04/2020 18:05, Craig Barratt via BackupPC-users ha scritto: What version of BackupPC are you running? 4.x will likely be a good deal faster than 3.x for both rsync+ssh and rsyncd. I have 3.2.1 backing up this imap folder. I'm evaluating whether to upgrade to V4 but I am mirroring some pc/ folders on a another idle backuppc installation and this is not feasible with V4 at this moment. Thank you Raf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files
I would expect no difference for small file performance between rsync and ssh-rsync. The ssh overhead on a modern system limits data rate to something like 75MB/s, nearly saturating a gigabit link. It seems you have basic filesystem performance issues. More RAM, larger caches, SSD? investigate with iostat on client or server? ED. > On 2020, Apr 21, at 4:31 AM, R.C. wrote: > > Hi > > What is the expected difference in performance between rsync+shh and rsyncd? > I would use it over a private LAN, so no concerns about security. > Currently rsync+ssh is way too slow for a huge number of very small files > (about 700K email files in an imap server tree), even without --checksum. > > Thank you > > Raf > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync vs rsyncd speed for huge number of small files
What version of BackupPC are you running? 4.x will likely be a good deal faster than 3.x for both rsync+ssh and rsyncd. The penalty of rsync+ssh vs rsyncd is likely modest, although it depends on how much data is changing between backups. Craig On Tue, Apr 21, 2020 at 1:33 AM R.C. wrote: > Hi > > What is the expected difference in performance between rsync+shh and > rsyncd? > I would use it over a private LAN, so no concerns about security. > Currently rsync+ssh is way too slow for a huge number of very small > files (about 700K email files in an imap server tree), even without > --checksum. > > Thank you > > Raf > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] rsync vs rsyncd speed for huge number of small files
Hi What is the expected difference in performance between rsync+shh and rsyncd? I would use it over a private LAN, so no concerns about security. Currently rsync+ssh is way too slow for a huge number of very small files (about 700K email files in an imap server tree), even without --checksum. Thank you Raf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
