Re: [BackupPC-users] Slow transfer via rsync?
Hi Holger& all, >> [...] >> Why is it still so slow? > it's been mentioned before, though not explicitly. What is the RTT over the > VPN (i.e. between BackupPC server and client host)? So here are several outputs: [root@bu ~]# cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) [root@bu ~]# uname -r 3.10.0-229.14.1.el7.x86_64 [root@bu ~]# rsync --version rsync version 3.0.9 protocol version 30 [root@bu ~]# ps ax |grep infra2 21811 ?S 1:41 /usr/bin/perl /usr/share/BackupPC/bin/BackupPC_dump infra2.domain.de 21814 ?S 0:01 /usr/bin/ssh -l backup infra2.domain.de /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive -x --checksum-seed=32761 --ignore-times . /srv/ 21835 ?S 0:16 /usr/bin/perl /usr/share/BackupPC/bin/BackupPC_dump infra2.domain.de traceroute to 192.168.1.3 (192.168.1.3), 30 hops max, 60 byte packets 1 10.12.0.1 (10.12.0.1) 344.955 ms 355.805 ms 355.818 ms 2 infra2 (192.168.1.3) 359.722 ms 359.737 ms 364.983 ms PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_seq=1 ttl=63 time=256 ms 64 bytes from 192.168.1.3: icmp_seq=2 ttl=63 time=345 ms 64 bytes from 192.168.1.3: icmp_seq=3 ttl=63 time=374 ms 64 bytes from 192.168.1.3: icmp_seq=4 ttl=63 time=424 ms top on the server while a backup to this client is running %Cpu(s): 1,8 us, 0,6 sy, 0,0 ni, 95,6 id, 0,9 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem : 24515984 total, 189184 free, 1219728 used, 23107072 buff/cache KiB Swap: 12582908 total, 12257544 free, 325364 used. 23017216 avail Mem tcpdump -i tun0 host 192.168.1.3 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes 21:25:46.084584 IP infra2.ssh > bu.54020: Flags [.], seq 3849499977:3849501317, ack 3346469010, win 3755, options [nop,nop,TS val 844038020 ecr 275484056], length 1340 21:25:46.084604 IP bu.54020 > infra2.ssh: Flags [.], ack 4294965956, win 450, options [nop,nop,TS val 275484429 ecr 844037551,nop,nop,sack 1 {0:1340}], length 0 21:25:46.433176 IP infra2.ssh > bu.54020: Flags [.], seq 4294965956:0, ack 1, win 3755, options [nop,nop,TS val 844038396 ecr 275484429], length 1340 21:25:46.433206 IP bu.54020 > infra2.ssh: Flags [.], ack 1340, win 441, options [nop,nop,TS val 275484778 ecr 844038396], length 0 21:25:46.735195 IP infra2.ssh > bu.54020: Flags [.], seq 1340:2680, ack 1, win 3755, options [nop,nop,TS val 844038742 ecr 275484778], length 1340 21:25:46.735216 IP bu.54020 > infra2.ssh: Flags [.], ack 2680, win 450, options [nop,nop,TS val 275485080 ecr 844038742], length 0 21:25:47.025229 IP infra2.ssh > bu.54020: Flags [.], seq 2680:4020, ack 1, win 3755, options [nop,nop,TS val 844039046 ecr 275485080], length 1340 21:25:47.025249 IP bu.54020 > infra2.ssh: Flags [.], ack 4020, win 450, options [nop,nop,TS val 275485370 ecr 844039046], length 0 21:25:47.051452 IP infra2.ssh > bu.54020: Flags [.], seq 4020:5360, ack 1, win 3755, options [nop,nop,TS val 844039046 ecr 275485080], length 1340 21:25:47.051471 IP bu.54020 > infra2.ssh: Flags [.], ack 5360, win 450, options [nop,nop,TS val 275485396 ecr 844039046], length 0 21:25:47.361984 IP infra2.ssh > bu.54020: Flags [.], seq 5360:6700, ack 1, win 3755, options [nop,nop,TS val 844039337 ecr 275485370], length 1340 21:25:47.362017 IP bu.54020 > infra2.ssh: Flags [.], ack 6700, win 450, options [nop,nop,TS val 275485706 ecr 844039337], length 0 21:25:47.390806 IP infra2.ssh > bu.54020: Flags [.], seq 6700:8040, ack 1, win 3755, options [nop,nop,TS val 844039337 ecr 275485370], length 1340 21:25:47.390826 IP bu.54020 > infra2.ssh: Flags [.], ack 8040, win 450, options [nop,nop,TS val 275485735 ecr 844039337], length 0 21:25:47.420871 IP infra2.ssh > bu.54020: Flags [.], seq 8040:9380, ack 1, win 3755, options [nop,nop,TS val 844039361 ecr 275485396], length 1340 21:25:47.420890 IP bu.54020 > infra2.ssh: Flags [.], ack 9380, win 450, options [nop,nop,TS val 275485765 ecr 844039361], length 0 ^C 16 packets captured 16 packets received by filter 0 packets dropped by kernel I am monitoring the server through SNMP and I can see the tun0 interface is used by 120kb/s inbound and 4k/s outbound. Outbound spiked to 130kb/s for the first 15minutes of the backup to this client. Besides of the backup there is nothing else going on on the
Re: [BackupPC-users] Slow transfer via rsync?
Hi, Christian Völker wrote on 2015-09-28 14:55:35 +0200 [Re: [BackupPC-users] Slow transfer via rsync?]: > [...] > Why is it still so slow? it's been mentioned before, though not explicitly. What is the RTT over the VPN (i.e. between BackupPC server and client host)? How many files are we talking about? Is the firewall blocking (any) ICMP traffic between the two endpoints? A share "/srv" suggests this probably isn't Windoze ;-). Is there anything special about this client host (kernel version, distribution release, ...)? Can you use tcpdump to look at the characteristics of the connection (e.g. is there a constant (slow) exchange, or are there hangs, timeouts, retransmissions)? Is your OpenVPN connection over TCP or UDP? Tunnelling TCP over TCP might give such problems, depending on the characteristics of the underlying network (e.g. significant packet loss/congestion). All of that said, the *first* backup of a host is usually special in that all data not already in the pool (which might in this case mean all data) needs to be compressed. The *second* backup may add rsync checksums. All of this might show up as CPU usage, I/O waits, etc., but it might also just add some more latency to the network latency. If your first backup has been running for 24 hours, that is merely twice the time a quick best case estimate gives. I'd be annoyed, but I'd give it some more time before starting to worry. Granted, the failing backups on the old host *are* something to worry about. ClientTimeout? Also, the network usage you are seeing is way too low and seems to indicate that you will reach the point when worrying is warranted ;-). Regards, Holger -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow transfer via rsync?
Hi B, Most probably, an electrical glitch messed it up. So, may be a reboot could fix your problem as well. Well, I can obviously reboot. But it is a virtual machine running on a VMware ESXi server I doubt it will help. Uptime is currently at 59days which means the issue with the slow backup startet months before the last reboot happened. Anyway, nice idea ;) /Christian -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow transfer via rsync?
And what about your backupPC server ? Check it for memory usage. I fall in such problems when adding a new host with large dataset to backup (> 100GB, all previous shares where near 10GB each) and the problem was related to the available RAM on my server. As it was a virtual backuppc server I increase the allowed memory and it solves the problem. The symptoms were the same than yours, few cpu usage, few network bandwith used and very slow backup. Patrick Christian Völker wrote: > Hi guys, > > I am using BackupPC now for years and it was always running perfectly. > Around 25 hosts; /var/lib/BackupPC is around 2TB). > > Currently I am having an issue with one of my remote hosts. > > I even created a new instance of BackupPC to test and see what happens- > but it is the same! > > > The remote host has two share to be backed up. / with around 4GB of data > and /srv with 27GB of data. > > The connection is now a 5Mbit/s (uplink) leased line with fixed IP > (connected through OpenVPN to the BackuPC host). The backup takes ages! > > When calculating with a 5Mbit/s linke 27GB should be transferred within > ~15hours. Here, the initial full backup is not yet done and is already > running more than 24hours! > > I can see some sort of progress if I do a "lsof| grep rsync" on the > client. But it still appears to be ways to slow! On the client "top" > tells me 99% idle CPU (and no I/O waits). The NIC and the Internet > connection has an outbound usage of around 350kb/s- far away from the > 5Mbit/s the line offers. > And yes, there is no other outbound traffic going on. I just did a > Ms-RDP session and the line usage goes up to the expected 2.5Mbit/s! > > Do you have a clue why it takes to transfer? > > It does not seem to be linkes to a new full or incr backup- my existing > backup server was not able to back up the machine now for 120days! Even > though it tells me 24/7 "backup in progress". > > So, any idea why this performs so badly? > > Greetings > > Christian > > > -- > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > -- === | Equipe M.O.S.T. | | | Patrick BEGOU | mailto:patrick.be...@grenoble-inp.fr | | LEGI| | | BP 53 X | Tel 04 76 82 51 35 | | 38041 GRENOBLE CEDEX| Fax 04 76 82 52 71 | === -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow transfer via rsync?
On Mon, 28 Sep 2015 05:41:31 +0200 Christian Völkerwrote: > >> When calculating with a 5Mbit/s linke 27GB should be transferred > >> within ~15hours. Here, the initial full backup is not yet done and > >> is already running more than 24hours! > > Make sure you only use one compression order in any place (either > > backuppc or openvpn, NOT both). > How should this do any harm? Simply because trying to compress an already compressed stream make it bigger. > The router is a totally different pc. See above. > Shouldn't the cpu usage be high at least if OpenVPN settings would > interfere here? Depends on CPU power; do as Patrick suggested: check your backuppc svr RAM (& CPU) usage. > > And as you have a fixed IP address, why not simply use a regular SSH > > connection instead of openvpn? (I've seen it ridiculed by a simple > > remote desktop connection on w$…) > What do you mean? I do not want to use direct connections as I have to > do a port forward in this case. So, you have a Ferrari in the garage but keep using an old sedan… Sweet memories or to pass the time somehow? ;-p) JY -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow transfer via rsync?
Hi, Christian Völker wrote on 2015-09-28 22:02:26 +0200 [Re: [BackupPC-users] Slow transfer via rsync?]: > >> Why is it still so slow? > [...] > traceroute to 192.168.1.3 (192.168.1.3), 30 hops max, 60 byte packets > 1 10.12.0.1 (10.12.0.1) 344.955 ms 355.805 ms 355.818 ms > 2 infra2 (192.168.1.3) 359.722 ms 359.737 ms 364.983 ms are you serious? Where do you get over 1/3 second delay from? I'd expect about a tenth of that (just tested over a UDP OpenVPN link). This is where I'd look. It obviously *is* latency related. What do you get from a traceroute from one VPN endpoint to the other (not through the VPN)? Oh, is this traceroute in parallel to the running rsync exchange? Not that it should make much difference, considering the low bandwidth usage ... > [...] > > Is the firewall blocking (any) ICMP traffic between the two endpoints? > No. As you can see in the above commands it appears everything if fine. No, I don't really see that. I see that ICMP echo request/reply are working, not that PMTU discovery is working (but maybe I missed it). > > Can you use tcpdump to look at the characteristics > > of the connection (e.g. is there a constant (slow) exchange, or are there > > hangs, timeouts, retransmissions)? > See above- there is an exchange but I am unsure how to read the output > of tcpdump in detail. It wasn't about the detail, it was about the timing. The timestamps seem to indicate a rhythm corresponding to the RTT with something like 1340 bytes of [raw IP] data sent every 300ms - faster at the end (because more is sent before the ACK arrives). The fragment you quoted seemed to show data transfer only from "infra2" to "bu" (and ACKs without data in the other direction), so - at this point - it doesn't seem to be the rsync protocol exchange limiting the speed. It also doesn't *appear* to be the TCP window size [yet], because the last three packets from infra2 to bu are sent without waiting for the ACK whereas the first ones apparently aren't. With an RTT of around about 333 ms, you would need a window size of 213KB for a theoretical throughput of 5 Mbit/s (assuming data is only transfered in one direction without waiting for rsync protocol responses from the other side). Please avoid line wrapping in future tcpdump quotes. > [...] > UDP for OpenVPN: > [...] > tls-client The server side could have more options that affect speed ... > > [...] the *first* backup of a host is usually special in that all data > > not already in the pool (which might in this case mean all data) needs to > > be compressed. > Prior to be compressed it should have been transferred. Does BackuPC > compress "on-the-fly" or after all data is transferred? However- CPU > cycles are close to zero (see above). On the fly. The bandwidth of the data stream is not high enough to make compression use significant CPU resources, but it *will* take *some* time, which *might* add another few (tens of) ms to a RTT if the File::RsyncP implementation compresses data before sending a protocol reply the remote end is waiting for. I don't know the protocol exchange well enough to know whether this will have a measurable effect (if any). > > [...] I'd give it some more time before starting to worry. > I wouldn't worry if I at least one of the parameters (IO, CPU, memory, > network) would show some usage. But none of them really does. It's about latency, not about saturation. Regards, Holger -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow transfer via rsync?
On Mon, Sep 28, 2015 at 10:17:13PM +0200, Christian Völker wrote: > I doubt it will help. Uptime is currently at 59days which means the > issue with the slow backup startet months before the last reboot happened. > > Anyway, nice idea ;) Have you a simple `rsync --progress -avr host:module/ /tmp/' at a shell? If it acts the same then you've eliminated BackupPC completely from the problem. -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/