It doesn't sound like rdiff-backup is the culprit here. You could try hpn-ssh https://sourceforge.net/projects/hpnssh/ ?
On 29 March 2016 at 21:03, Derek Atkins <de...@ihtfp.com> wrote: > Hi, > > On Mon, March 28, 2016 10:56 am, Derek Atkins wrote: > > Hi, > > > > Thank you for taking the time to look at this.. > > > > On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote: > >> Is this really your first rdiff-backup to this location? If you have any > >> previous rdiff-backup runs to this repository then the situation is > >> complicated by rdiff-backup's need to create a new set of reverse diff > >> files to be able to regress to previous file contents. > > > > Yes, this s really the first rdiff-backup to this location. > > > > A second backup run shortly after the first one completed finished in 55 > > minutes. > > > >> What is your /tmp location? rdiff-backup uses this location for some > >> operations though not AFAIK for standard backup runs. Still, if /tmp is > >> on > >> encfs maybe it could be a culprit; you can override rdiff-backup's > >> temporary file location with --tempdir and --remote-tempdir. > > > > If it is truly /tmp then no; /tmp is a ramdisk on the backup server and > is > > on the root disk on the target server. Neither are being run through > > encfs. > > > > If, however, you mean the rdiff-backup-data.tmp files, those ARE being > run > > through encfs. > > > >> Might also be worth trying --ssh-no-compression. > > > > I already have "Compression no" set in ~/.ssh/config so I'm not sure what > > this would add? > > > >> Dominic > >> http://www.timedicer.co.uk > > > > -derek > > > > PS: I'm running a raw rsync command now just to see how it behaves -- so > > far I'm only seeing about 2MB/s, but it's only been running for 10 > minutes > > or so. > > My rsync backup just finished. It copied 202688912 KB in 1599m55.255s so > about 2.1MB/s. Still significantly slower than SCP, but faster than > rdiff-backup. > > The command I ran was: > > rsync -art -e "ssh -l root -i /dev/null -o Compression=no" > root@server:/var/www/ /backups/server > > :-( > > -derek > > > > >> > >> On 28 March 2016 at 14:37, Derek Atkins <de...@ihtfp.com> wrote: > >> > >>> Just a quick update: I tried making these changes on both sides and it > >>> really didn't make a difference. Full backup of 202852072 Kbytes > >>> required 2267m25.913s (previously it took 2346m57.800s, so it only sped > >>> up a factor of 3%. > >>> > >>> Only thing I have not yet tried is running a raw rsync to see how fast > >>> that runs. I'll do that next. > >>> > >>> So, back to my orignal question: anyone have any idea how to get > >>> initial > >>> transfers to run faster (or indeed any significant data transfers)? > >>> > >>> Thanks, > >>> > >>> -derek > >>> > >>> "Derek Atkins" <de...@ihtfp.com> writes: > >>> > >>> > Hi, > >>> > > >>> > I'm trying to use rdiff-backup to backup a bunch of servers. One > >>> > particular server contains about 160GB of data, but when I try to > >>> perform > >>> > the rdiff-backup it's saving the data at a measly 1MB/s. > >>> > > >>> > Here's my configuration: > >>> > > >>> > [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas] > >>> > > >>> > I ran a bunch of tests to try to figure out my bottlenecks. > >>> > > >>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on > >>> the > >>> > backup server. Going directly to FreeNAS via NFS (bybassing encfs) I > >>> get > >>> > 50.2MB/s. If I run dd directly on the backup server (through encfs) > >>> I > >>> get > >>> > 20.1MB/s. If I go over SSH from the backup server to the target > >>> server > >>> > and run the dd on the target server, then write to FreeNas through > >>> encfs > >>> > declines to 7.6MB/s. > >>> > > >>> > Note that in those SSH tests, however, I forgot to turn off > >>> compression. > >>> > When I do that, the throughput for the dd test reduced to 6.6BM/s. > >>> (Note > >>> > that this is running simultaneously with a running rdiff-backup, so > >>> it's > >>> > possible that they are reducing performance). > >>> > > >>> > Then I ran an scp test to the same target server; copying about 1.4GB > >>> of > >>> > photos. Files ranged in size from 10KB to 5MB. When run in standard > >>> mode > >>> > (displaying each file status) I got 4.4MB/s. Running in quiet mode I > >>> get > >>> > 5.1MB/s. > >>> > > >>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO) > >>> should > >>> > not be significantly slower than the last dd-over-ssh test. It > >>> appears > >>> > rdiff-backup is slowing me down by a factor of 5x throughput versus > >>> scp. > >>> > > >>> > I found a message from Ben from 2005 where he suggests increasing the > >>> > blocksize and conn_bufsiz settings in Globals.py: > >>> > > >>> > https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html > >>> > > >>> > What he didn't say was whether this needed to be changed on the > >>> target > >>> > server, the backup server, or both. Nor do I know if that would > >>> actually > >>> > help this situation. > >>> > > >>> > Do you have any ideas? > >>> > > >>> > Thanks, > >>> > > >>> > -derek > >>> > > >>> > PS: According to rpm, both systems are running version 1.2.8. > >>> > >>> -- > >>> Derek Atkins 617-623-3745 > >>> de...@ihtfp.com www.ihtfp.com > >>> Computer and Internet Security Consultant > >>> > >>> _______________________________________________ > >>> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org > >>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users > >>> Wiki URL: > >>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki > >>> > >> > > > > > > -- > > Derek Atkins 617-623-3745 > > de...@ihtfp.com www.ihtfp.com > > Computer and Internet Security Consultant > > > > > > _______________________________________________ > > rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org > > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users > > Wiki URL: > > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki > > > > > -- > Derek Atkins 617-623-3745 > de...@ihtfp.com www.ihtfp.com > Computer and Internet Security Consultant > >
_______________________________________________ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
