Re: [BackupPC-users] Slow transfer via rsync?

[Christian Völker]

Hi Holger& all,

>> [...]
>> Why is it still so slow?
> it's been mentioned before, though not explicitly. What is the RTT over the
> VPN (i.e. between BackupPC server and client host)?

So here are several outputs:

[root@bu ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@bu ~]# uname -r
3.10.0-229.14.1.el7.x86_64
[root@bu ~]# rsync --version
rsync  version 3.0.9  protocol version 30

[root@bu ~]# ps ax |grep infra2
21811 ?S  1:41 /usr/bin/perl
/usr/share/BackupPC/bin/BackupPC_dump infra2.domain.de
21814 ?S  0:01 /usr/bin/ssh -l backup infra2.domain.de
/usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms
--owner --group -D --links --hard-links --times --block-size=2048
--recursive -x --checksum-seed=32761 --ignore-times . /srv/
21835 ?S  0:16 /usr/bin/perl
/usr/share/BackupPC/bin/BackupPC_dump infra2.domain.de

traceroute to 192.168.1.3 (192.168.1.3), 30 hops max, 60 byte packets
 1  10.12.0.1 (10.12.0.1)  344.955 ms  355.805 ms  355.818 ms
 2  infra2 (192.168.1.3)  359.722 ms  359.737 ms  364.983 ms

PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=63 time=256 ms
64 bytes from 192.168.1.3: icmp_seq=2 ttl=63 time=345 ms
64 bytes from 192.168.1.3: icmp_seq=3 ttl=63 time=374 ms
64 bytes from 192.168.1.3: icmp_seq=4 ttl=63 time=424 ms

top on the server while a backup to this client is running
%Cpu(s): 1,8 us,  0,6 sy,  0,0 ni, 95,6 id,  0,9 wa,  0,0 hi,  0,1 si, 
0,0 st
KiB Mem : 24515984 total,   189184 free,  1219728 used, 23107072 buff/cache
KiB Swap: 12582908 total, 12257544 free,   325364 used. 23017216 avail Mem

tcpdump -i tun0 host 192.168.1.3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
21:25:46.084584 IP infra2.ssh > bu.54020: Flags [.], seq
3849499977:3849501317, ack 3346469010, win 3755, options [nop,nop,TS val
844038020 ecr 275484056], length 1340
21:25:46.084604 IP bu.54020 > infra2.ssh: Flags [.], ack 4294965956, win
450, options [nop,nop,TS val 275484429 ecr 844037551,nop,nop,sack 1
{0:1340}], length 0
21:25:46.433176 IP infra2.ssh > bu.54020: Flags [.], seq 4294965956:0,
ack 1, win 3755, options [nop,nop,TS val 844038396 ecr 275484429],
length 1340
21:25:46.433206 IP bu.54020 > infra2.ssh: Flags [.], ack 1340, win 441,
options [nop,nop,TS val 275484778 ecr 844038396], length 0
21:25:46.735195 IP infra2.ssh > bu.54020: Flags [.], seq 1340:2680, ack
1, win 3755, options [nop,nop,TS val 844038742 ecr 275484778], length 1340
21:25:46.735216 IP bu.54020 > infra2.ssh: Flags [.], ack 2680, win 450,
options [nop,nop,TS val 275485080 ecr 844038742], length 0
21:25:47.025229 IP infra2.ssh > bu.54020: Flags [.], seq 2680:4020, ack
1, win 3755, options [nop,nop,TS val 844039046 ecr 275485080], length 1340
21:25:47.025249 IP bu.54020 > infra2.ssh: Flags [.], ack 4020, win 450,
options [nop,nop,TS val 275485370 ecr 844039046], length 0
21:25:47.051452 IP infra2.ssh > bu.54020: Flags [.], seq 4020:5360, ack
1, win 3755, options [nop,nop,TS val 844039046 ecr 275485080], length 1340
21:25:47.051471 IP bu.54020 > infra2.ssh: Flags [.], ack 5360, win 450,
options [nop,nop,TS val 275485396 ecr 844039046], length 0
21:25:47.361984 IP infra2.ssh > bu.54020: Flags [.], seq 5360:6700, ack
1, win 3755, options [nop,nop,TS val 844039337 ecr 275485370], length 1340
21:25:47.362017 IP bu.54020 > infra2.ssh: Flags [.], ack 6700, win 450,
options [nop,nop,TS val 275485706 ecr 844039337], length 0
21:25:47.390806 IP infra2.ssh > bu.54020: Flags [.], seq 6700:8040, ack
1, win 3755, options [nop,nop,TS val 844039337 ecr 275485370], length 1340
21:25:47.390826 IP bu.54020 > infra2.ssh: Flags [.], ack 8040, win 450,
options [nop,nop,TS val 275485735 ecr 844039337], length 0
21:25:47.420871 IP infra2.ssh > bu.54020: Flags [.], seq 8040:9380, ack
1, win 3755, options [nop,nop,TS val 844039361 ecr 275485396], length 1340
21:25:47.420890 IP bu.54020 > infra2.ssh: Flags [.], ack 9380, win 450,
options [nop,nop,TS val 275485765 ecr 844039361], length 0
^C
16 packets captured
16 packets received by filter
0 packets dropped by kernel

I am monitoring the server through SNMP and I can see the tun0 interface
is used by 120kb/s inbound and 4k/s outbound. Outbound spiked to 130kb/s
for the first 15minutes of the backup to this client. Besides of the
backup there is nothing else going on on the 

Re: [BackupPC-users] Slow transfer via rsync?

[Holger Parplies]

Hi,

Christian Völker wrote on 2015-09-28 14:55:35 +0200 [Re: [BackupPC-users] Slow 
transfer via rsync?]:
> [...]
> Why is it still so slow?

it's been mentioned before, though not explicitly. What is the RTT over the
VPN (i.e. between BackupPC server and client host)?

How many files are we talking about? Is the firewall blocking (any) ICMP
traffic between the two endpoints? A share "/srv" suggests this probably isn't
Windoze ;-). Is there anything special about this client host (kernel version,
distribution release, ...)? Can you use tcpdump to look at the characteristics
of the connection (e.g. is there a constant (slow) exchange, or are there
hangs, timeouts, retransmissions)?

Is your OpenVPN connection over TCP or UDP? Tunnelling TCP over TCP might give
such problems, depending on the characteristics of the underlying network
(e.g. significant packet loss/congestion).


All of that said, the *first* backup of a host is usually special in that all
data not already in the pool (which might in this case mean all data) needs to
be compressed. The *second* backup may add rsync checksums. All of this might
show up as CPU usage, I/O waits, etc., but it might also just add some more
latency to the network latency. If your first backup has been running for 24
hours, that is merely twice the time a quick best case estimate gives. I'd be
annoyed, but I'd give it some more time before starting to worry. Granted, the
failing backups on the old host *are* something to worry about. ClientTimeout?
Also, the network usage you are seeing is way too low and seems to indicate
that you will reach the point when worrying is warranted ;-).

Regards,
Holger

--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Slow transfer via rsync?

[Christian Völker]

Hi B,

Most probably, an electrical glitch messed it up.

So, may be a reboot could fix your problem as well.

Well, I can obviously reboot. But it is a virtual machine running on a
VMware ESXi server

I doubt it will help. Uptime is currently at 59days which means the
issue with the slow backup startet months before the last reboot happened.

Anyway, nice idea ;)

/Christian

--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Slow transfer via rsync?

[Patrick Begou]

And what about your backupPC server ? Check it for memory usage.
I fall in such problems when adding a new host with large dataset to backup (> 
100GB, all previous shares where near 10GB each) and the problem was related to 
the available RAM on my server. As it was a virtual backuppc server I increase 
the allowed memory and it solves the problem.
The symptoms were the same than yours, few cpu usage, few network bandwith used 
and very slow backup.

Patrick

Christian Völker wrote:
> Hi guys,
>
> I am using BackupPC now for years and it was always running perfectly.
> Around 25 hosts; /var/lib/BackupPC is around 2TB).
>
> Currently I am having an issue with one of my remote hosts.
>
> I even created a new instance of BackupPC to test and see what happens-
> but it is the same!
>
>
> The remote host has two share to be backed up. / with around 4GB of data
> and /srv with 27GB of data.
>
> The connection is now a 5Mbit/s (uplink) leased line with fixed IP
> (connected through OpenVPN to the BackuPC host).  The backup takes ages!
>
> When calculating with a 5Mbit/s linke 27GB should be transferred within
> ~15hours. Here, the initial full backup is not yet done and is already
> running more than 24hours!
>
> I can see some sort of progress if I do a "lsof| grep rsync" on the
> client. But it  still appears to be ways to slow! On the client "top"
> tells me 99% idle CPU (and no I/O waits). The NIC and the Internet
> connection has an outbound usage of around 350kb/s- far away from the
> 5Mbit/s the line offers.
> And yes, there is no other outbound traffic going on. I just did a
> Ms-RDP session and the line usage goes up to the expected 2.5Mbit/s!
>
> Do you have a clue why it takes to transfer?
>
> It does not seem to be linkes to a new full or incr backup- my existing
> backup server was not able to back up the machine now for 120days! Even
> though it tells me 24/7 "backup in progress".
>
> So, any idea why this performs so badly?
>
> Greetings
>
> Christian
>
>
> --
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>


-- 
===
|  Equipe M.O.S.T. |  |
|  Patrick BEGOU   | mailto:patrick.be...@grenoble-inp.fr |
|  LEGI|  |
|  BP 53 X | Tel 04 76 82 51 35   |
|  38041 GRENOBLE CEDEX| Fax 04 76 82 52 71   |
===



--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Slow transfer via rsync?

[Bzzzz]

On Mon, 28 Sep 2015 05:41:31 +0200
Christian Völker  wrote:

> >> When calculating with a 5Mbit/s linke 27GB should be transferred
> >> within ~15hours. Here, the initial full backup is not yet done and
> >> is already running more than 24hours!
> > Make sure you only use one compression order in any place (either
> > backuppc or openvpn, NOT both).

> How should this do any harm?

Simply because trying to compress an already compressed stream make it
bigger.

> The router is a totally different pc.

See above.

> Shouldn't the cpu usage be high at least if OpenVPN settings would
> interfere here?

Depends on CPU power; do as Patrick suggested: check your backuppc svr
RAM (& CPU) usage.
 
> > And as you have a fixed IP address, why not simply use a regular SSH
> > connection instead of openvpn? (I've seen it ridiculed by a simple
> > remote desktop connection on w$…)

> What do you mean? I do not want to use direct connections as I have to
> do a port forward in this case.

So, you have a Ferrari in the garage but keep using an old sedan…
Sweet memories or to pass the time somehow? ;-p)

JY

--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Slow transfer via rsync?

[Holger Parplies]

Hi,

Christian Völker wrote on 2015-09-28 22:02:26 +0200 [Re: [BackupPC-users] Slow 
transfer via rsync?]:
> >> Why is it still so slow?
> [...]
> traceroute to 192.168.1.3 (192.168.1.3), 30 hops max, 60 byte packets
>  1  10.12.0.1 (10.12.0.1)  344.955 ms  355.805 ms  355.818 ms
>  2  infra2 (192.168.1.3)  359.722 ms  359.737 ms  364.983 ms

are you serious? Where do you get over 1/3 second delay from? I'd expect about
a tenth of that (just tested over a UDP OpenVPN link). This is where I'd look.
It obviously *is* latency related. What do you get from a traceroute from one
VPN endpoint to the other (not through the VPN)?

Oh, is this traceroute in parallel to the running rsync exchange? Not that it
should make much difference, considering the low bandwidth usage ...

> [...]
> >  Is the firewall blocking (any) ICMP traffic between the two endpoints? 
> No. As you can see in the above commands it appears everything if fine.

No, I don't really see that. I see that ICMP echo request/reply are working,
not that PMTU discovery is working (but maybe I missed it).

> > Can you use tcpdump to look at the characteristics
> > of the connection (e.g. is there a constant (slow) exchange, or are there
> > hangs, timeouts, retransmissions)?
> See above- there is an exchange but I am unsure how to read the output
> of tcpdump in detail.

It wasn't about the detail, it was about the timing. The timestamps seem to
indicate a rhythm corresponding to the RTT with something like 1340 bytes of
[raw IP] data sent every 300ms - faster at the end (because more is sent
before the ACK arrives). The fragment you quoted seemed to show data transfer
only from "infra2" to "bu" (and ACKs without data in the other direction), so
- at this point - it doesn't seem to be the rsync protocol exchange limiting
the speed. It also doesn't *appear* to be the TCP window size [yet], because
the last three packets from infra2 to bu are sent without waiting for the ACK
whereas the first ones apparently aren't.
With an RTT of around about 333 ms, you would need a window size of 213KB for
a theoretical throughput of 5 Mbit/s (assuming data is only transfered in one
direction without waiting for rsync protocol responses from the other side).

Please avoid line wrapping in future tcpdump quotes.

> [...]
> UDP for OpenVPN:
> [...]
> tls-client

The server side could have more options that affect speed ...

> > [...] the *first* backup of a host is usually special in that all data
> > not already in the pool (which might in this case mean all data) needs to
> > be compressed. 
> Prior to be compressed it should have been transferred. Does BackuPC
> compress "on-the-fly" or after all data is transferred?  However- CPU
> cycles are close to zero (see above).

On the fly. The bandwidth of the data stream is not high enough to make
compression use significant CPU resources, but it *will* take *some* time,
which *might* add another few (tens of) ms to a RTT if the File::RsyncP
implementation compresses data before sending a protocol reply the remote
end is waiting for. I don't know the protocol exchange well enough to know
whether this will have a measurable effect (if any).

> > [...] I'd give it some more time before starting to worry.
> I wouldn't worry if I at least one of the parameters (IO, CPU, memory,
> network) would show some usage. But none of them really does.

It's about latency, not about saturation.

Regards,
Holger

--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Slow transfer via rsync?

[ngharo]

On Mon, Sep 28, 2015 at 10:17:13PM +0200, Christian Völker wrote:
> I doubt it will help. Uptime is currently at 59days which means the
> issue with the slow backup startet months before the last reboot happened.
> 
> Anyway, nice idea ;)

Have you a simple `rsync --progress -avr host:module/ /tmp/' at a shell?

If it acts the same then you've eliminated BackupPC completely from the
problem.

--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] Slow transfer via rsync?

[Christian Völker]

Hi guys,

I am using BackupPC now for years and it was always running perfectly.
Around 25 hosts; /var/lib/BackupPC is around 2TB).

Currently I am having an issue with one of my remote hosts.

I even created a new instance of BackupPC to test and see what happens-
but it is the same!


The remote host has two share to be backed up. / with around 4GB of data
and /srv with 27GB of data.

The connection is now a 5Mbit/s (uplink) leased line with fixed IP
(connected through OpenVPN to the BackuPC host).  The backup takes ages!

When calculating with a 5Mbit/s linke 27GB should be transferred within
~15hours. Here, the initial full backup is not yet done and is already
running more than 24hours!

I can see some sort of progress if I do a "lsof| grep rsync" on the
client. But it  still appears to be ways to slow! On the client "top"
tells me 99% idle CPU (and no I/O waits). The NIC and the Internet
connection has an outbound usage of around 350kb/s- far away from the
5Mbit/s the line offers.
And yes, there is no other outbound traffic going on. I just did a
Ms-RDP session and the line usage goes up to the expected 2.5Mbit/s!

Do you have a clue why it takes to transfer?

It does not seem to be linkes to a new full or incr backup- my existing
backup server was not able to back up the machine now for 120days! Even
though it tells me 24/7 "backup in progress".

So, any idea why this performs so badly?

Greetings

Christian


--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Slow transfer via rsync?

[Bzzzz]

On Sun, 27 Sep 2015 22:48:43 +0200
Christian Völker  wrote:

> Hi guys,

Hi Chris,

…
> The connection is now a 5Mbit/s (uplink) leased line with fixed IP
> (connected through OpenVPN to the BackuPC host).  The backup takes
> ages!

How much is the downlink? (rsync needs a good deal of bi-dir exchanges)
 
> When calculating with a 5Mbit/s linke 27GB should be transferred within
> ~15hours. Here, the initial full backup is not yet done and is already
> running more than 24hours!

Make sure you only use one compression order in any place (either
backuppc or openvpn, NOT both).

And as you have a fixed IP address, why not simply use a regular SSH
connection instead of openvpn? (I've seen it ridiculed by a simple
remote desktop connection on w$…)

Jean-Yves

--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Slow transfer via rsync?

[Christian Völker]

Hi,
>> The connection is now a 5Mbit/s (uplink) leased line with fixed IP
>> (connected through OpenVPN to the BackuPC host).  The backup takes
>> ages!
> How much is the downlink? (rsync needs a good deal of bi-dir exchanges)
>  
25Mbit/s. The connection is for sure not the limiting factor. Uplinks is
used for ~350Kb/s while the downlink even uses less during the "backup".

>> When calculating with a 5Mbit/s linke 27GB should be transferred within
>> ~15hours. Here, the initial full backup is not yet done and is already
>> running more than 24hours!
> Make sure you only use one compression order in any place (either
> backuppc or openvpn, NOT both).
How should this do any harm? The router is a totally different pc.
Shouldn't the cpu usage be high at least if OpenVPN settings would
interfere here?

> And as you have a fixed IP address, why not simply use a regular SSH
> connection instead of openvpn? (I've seen it ridiculed by a simple
> remote desktop connection on w$…)
What do you mean? I do not want to use direct connections as I have to
do a port forward in this case.

Greetings

Christian


--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/