Re: Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
Hi. my last message was hard to read because of "sneaky" linebreaks that found there way into the mail when copying from text editor. I'm resending this message for better readability in the archive. Sorry about that. Berry --- Hi again. After fiddeling with pf and trying to statistically determine the cause of the problem i did another search on the net and found this thread [1] here there the user suggests: "when this slow problem occurs, you must disable the checksum on the physical and virtual cards and restart the xenserver." Does anyone know what exactly is meant by that? I then searched in the CVS log of xnf development [2] and found this statement: "Emulated em(4) or re(4) drivers will take over if xnf(4) driver is disabled or not compiled in." This gave lead me to the idea to disable xnf in the kernel, for now just temporarily upon boot, for example like this [3]: boot> boot bsd -c ... ukc> disable xnf As a result the booted systems now finds the `vio` === Case 6: Client <= VPN = Server <= WAN (vio) * From client (10.8.0.99) download external file from WAN via VPN tunnel * with Virtio driver * Testresult: → curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 100M 100 100M0 0 2226k 0 0:00:46 0:00:46 --:--:-- 4041k If I'm not mistaken this gives reason to believe that there is an issue with the xnf driver. I will mostly like contact the xnf developer about this. In the best case xnf can be fixed and the download performance be improved. Thank you all. Berry [1] http://daemonforums.org/showthread.php?p=61158 [2] https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pv/if_xnf.c [3] https://man.openbsd.org/config
Re: Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
Hi again. After fiddling with pf and trying to statistically determine the cause of the problem I did another search on the net and found this thread [1]. Here the author suggests: "when this slow problem occurs, you must disable the checksum on the physical and virtual cards and restart the xenserver." Does anyone know what exactly is meant by that? I then searched in the CVS log of xnf development [2] and found this statement: "Emulated em(4) or re(4) drivers will take over if xnf(4) driver is disabled or not compiled in."
Re: Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
On 2017-10-31 16:57, Berry Wendermouth wrote: > I will check again with the VPS provider that the interface of the > virtual machine is set to the correct value (virtio). These are the current VM interface settings (anonymized): vif = [ 'vifname=some-name, model=virtio-net, rate=100Mb/s, bridge=xenbr0.781, mac=00:x:x:x:x:x, ip=x.x.x.x x:x:0:0:0:0:3:7' ] So it seems OpenBSD 6.2 finds another way to set the network device, that is `xnf` instead of `vio`. Any thoughts on that?
Re: Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
On 2017-10-31 16:00, Chris Cappuccio wrote: > You went from emulated Realtek ethernet to xnf. Can you try other > network interfaces? How would I do this, isn't the interface auto detected by the kernel? So in the 6.1/i386 setup, the default interface was Realtek. We had speed problems with this interface and so we had the VPS provider change the interface of the virtual machine to Virtio, something like this: vif = [ 'vifname=name, model=virto, rate=100Mb/s, bridge=xenbr0.781, mac=00:50:00:00:00:00, ip=x.x.x.x x:x:0:0:0:0:3:7' ] This is also documented here [1][2]. After this change downloading via the external interface on the server worked very well. Now on 6.2 with the `xnf` interface, downloading directly on server also works fine. The problem only occours when download is requested from a VPN client. I will check again with the VPS provider that the interface of the virtual machine is set to the correct value (virtio). Thank you for your feedback. [1] http://xenbits.xen.org/docs/4.3-testing/misc/xl-network-configuration.html [2] https://wiki.xen.org/wiki/Virtio_On_Xen
Re: Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
> Per your request on #openbsd, I do a short reply, to let you reply to it > again... Thank you very much Kirill. > Have you tried to "download" from one of the clients, but without using > the VPN? You could use tcpbench or iperf in server mode on one of your > clients and do a port redirect from your WAN interface on the server to > a port which tcpbench or iperf is listening to. That way you can get > more clues regarding whether the issue is with OpenVPN or your network. The server can reach any client in subnet 10.8.0.0 only via VPN. However I noticed that I had a mistake in the iperf test 2 because I got confused with the direction data is send. As "man iperf" states: "To perform an iperf test the user must establish both a server (to discard traffic) and a client (to generate traffic)." Hence by default data is send from iperf client to server. This means in test case 2 data was send from VPN client 10.8.0.4 to VPN server 10.8.0.1, essentially testing upload speed. I conducted another test pushing data from external network to VPN client. === Case 4: WAN ==> Server = via VPN => Client * From some external node, send data to client via server via VPN tunnel * Testresults: # iperf -s -p 5002 Server listening on TCP port 5002 TCP window size: 85.3 KByte (default) [ 4] local 10.8.0.99 port 5002 connected with 85.x.x.x port 54230 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.8 sec 5.38 MBytes 4.19 Mbits/sec → iperf -c 109.x.x.x -p 5002 Client connecting to nohost.xyz, TCP port 5002 TCP window size: 45.0 KByte (default) [ 3] local 192.168.178.26 port 54230 connected with 109.x.x.x port 5002 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.5 sec 5.38 MBytes 4.27 Mbits/sec Compare this to the following: === Case 5: Client <= VPN = Server <= WAN * From client (10.8.0.99) download external file from WAN via VPN tunnel * Testresult: # curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 100M0 481690 0 4985 0 5:50:34 0:00:09 5:50:25 5055 So while pushing data from external network to vpn client works fine, downloading (requesting a download) from WAN on the client is very slow. Doesn't this imply that the VPN connection is "healthy" and that the problem is rather routing/firewall related? Cheers, Berry
Re: Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
You went from emulated Realtek ethernet to xnf. Can you try other network interfaces? Berry Wendermouth [bayb...@riseup.net] wrote: > Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after > upgrade > > > Dear OpenBSD Community, > > we are operating an OpenVPN server on OpenBSD. A few days ago we > upgraded to OpenBSD 6.2 > and we are now seeing very slow speeds (<10KB/s) when trying to download > via > the VPN tunnel from the internet (WAN). We did not have this problem > before. > > >From the documented test cases below (Specifically case 2) it does not > look like it is a VPN performance problem (e.g. mtu/encryption > performance related). > We can also exclude bandwidth trottleing by the VPS provider and the > ISP. > > * Did something essential change in `pf`? [4] > * Or is the problem related to OpenBSD's Xen drivers? > > Could someone help us track down the bottleneck? > > Any help and hints are very much appreciated. > > Thank you kindly > > Berry > > PS: for a better viewing experience you may compile this email body with > `asciidoc` > > == Environment > > === Server > * OpenBSD 6.2 / amd64 (-release) + syspatch > * OpenVPN 2.4.4 > * On Virtual Private Server / Xen version "4.9.0" by Xen Project [0] > * Detected CPU: Intel(R) Xeon(R) CPU E5-2620 > * Detected network device: xnf0 > * Firewall configuration: /etc/pf.conf [1] > * System Message Buffer [2] > > === Clients > * OpenBSD 6.2 with OpenVPN 2.4.4 > * GNU/Linux Gentoo with OpenVPN 2.4.4 > * LinesageOS 14.1 with OpenVPN for Android 0.6.73 > > == Detailed Problem Description / Test Results > > Please note: the following documented tests used one and the same client > / network connection: > > * GNU/Linux Gentoo with OpenVPN 2.4.4 > * Connected to router via wifi on internet connection with max 50Mbit/s > download > > To rule out problems with the client local network settings tests with > other client setups on other networks were also performed and showed > identical > results. For brevity they are not documented here. > > === Case 1: Server <==> WAN (ok) > * When on the server, downloading a file from WAN > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 10Mbit/s > * Testresult: > > > $ curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total% Received % Xferd Average Speed TimeTime Time > Current > Dload Upload Total SpentLeft Speed > 100 100M 100 100M0 0 9309k 0 0:00:11 0:00:11 --:--:-- > 10.9M > > > === Case 2: Client <= VPN => Server (ok) > * When on the client, downloading a file from server via VPN tunnel > * Scenario: standard download test with `iperf` > * Average Download Speed: ~ 15Mbit/s > * Testresult: > > > # iperf -s > > > --- > Server listening on TCP port 5001 > TCP window size: 16.0 KByte (default) > --- > [ 4] local 10.8.0.1 port 5001 connected with 10.8.0.4 port 34998 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.2 sec 18.5 MBytes 15.2 Mbits/sec > > > # iperf -c 10.8.0.1 > --- > Client connecting to 10.8.0.1, TCP port 5001 > TCP window size: 45.0 KByte (default) > --- > [ 3] local 10.8.0.4 port 34998 connected with 10.8.0.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 18.5 MBytes 15.5 Mbits/sec > > > === Case 3a: Client <= VPN => Server <==> WAN (broken) > * When on the client, downloading a file from WAN via VPN tunnel > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 5KB/s > * Testresult: > > > curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total% Received % Xferd Average Speed TimeTime Time > Current > Dload Upload Total SpentLeft Speed > 0 100M0 149k0 0 5102 0 5:42:32 0:00:30 5:42:02 > 4933 > > > === Case 3b: Client <==> WAN (ok) > * When on the client, downloading a file from WAN directly > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 1100KB/s > * Testresult: > > > curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total
Re: Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
Hi Per your request on #openbsd, I do a short reply, to let you reply to it again... * Berry Wendermouth [2017-10-30 10:48]: > Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after > upgrade > > > Dear OpenBSD Community, > > we are operating an OpenVPN server on OpenBSD. A few days ago we > upgraded to OpenBSD 6.2 > and we are now seeing very slow speeds (<10KB/s) when trying to download > via > the VPN tunnel from the internet (WAN). We did not have this problem > before. > > From the documented test cases below (Specifically case 2) it does not > look like it is a VPN performance problem (e.g. mtu/encryption > performance related). > We can also exclude bandwidth trottleing by the VPS provider and the > ISP. > > * Did something essential change in `pf`? [4] > * Or is the problem related to OpenBSD's Xen drivers? > > Could someone help us track down the bottleneck? > > Any help and hints are very much appreciated. Have you tried to "download" from one of the clients, but without using the VPN? You could use tcpbench or iperf in server mode on one of your clients and do a port redirect from your WAN interface on the server to a port which tcpbench or iperf is listening to. That way you can get more clues regarding whether the issue is with OpenVPN or your network. > Thank you kindly > > Berry > > PS: for a better viewing experience you may compile this email body with > `asciidoc` > > == Environment > > === Server > * OpenBSD 6.2 / amd64 (-release) + syspatch > * OpenVPN 2.4.4 > * On Virtual Private Server / Xen version "4.9.0" by Xen Project [0] > * Detected CPU: Intel(R) Xeon(R) CPU E5-2620 > * Detected network device: xnf0 > * Firewall configuration: /etc/pf.conf [1] > * System Message Buffer [2] > > === Clients > * OpenBSD 6.2 with OpenVPN 2.4.4 > * GNU/Linux Gentoo with OpenVPN 2.4.4 > * LinesageOS 14.1 with OpenVPN for Android 0.6.73 > > == Detailed Problem Description / Test Results > > Please note: the following documented tests used one and the same client > / network connection: > > * GNU/Linux Gentoo with OpenVPN 2.4.4 > * Connected to router via wifi on internet connection with max 50Mbit/s > download > > To rule out problems with the client local network settings tests with > other client setups on other networks were also performed and showed > identical > results. For brevity they are not documented here. > > === Case 1: Server <==> WAN (ok) > * When on the server, downloading a file from WAN > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 10Mbit/s > * Testresult: > > > $ curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total% Received % Xferd Average Speed TimeTime Time > Current > Dload Upload Total SpentLeft Speed > 100 100M 100 100M0 0 9309k 0 0:00:11 0:00:11 --:--:-- > 10.9M > > > === Case 2: Client <= VPN => Server (ok) > * When on the client, downloading a file from server via VPN tunnel > * Scenario: standard download test with `iperf` > * Average Download Speed: ~ 15Mbit/s > * Testresult: > > > # iperf -s > > > --- > Server listening on TCP port 5001 > TCP window size: 16.0 KByte (default) > --- > [ 4] local 10.8.0.1 port 5001 connected with 10.8.0.4 port 34998 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.2 sec 18.5 MBytes 15.2 Mbits/sec > > > # iperf -c 10.8.0.1 > --- > Client connecting to 10.8.0.1, TCP port 5001 > TCP window size: 45.0 KByte (default) > --- > [ 3] local 10.8.0.4 port 34998 connected with 10.8.0.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 18.5 MBytes 15.5 Mbits/sec > > > === Case 3a: Client <= VPN => Server <==> WAN (broken) > * When on the client, downloading a file from WAN via VPN tunnel > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 5KB/s > * Testresult: > > > curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total% Received % Xferd Average Speed TimeTime Time > Current > Dload Upload Total SpentLeft Speed > 0 100M0 149k0 0 5102 0 5:42:32 0:00:30 5:42:02 > 4933 > > > === Ca
Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade
Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after upgrade Dear OpenBSD Community, we are operating an OpenVPN server on OpenBSD. A few days ago we upgraded to OpenBSD 6.2 and we are now seeing very slow speeds (<10KB/s) when trying to download via the VPN tunnel from the internet (WAN). We did not have this problem before. >From the documented test cases below (Specifically case 2) it does not look like it is a VPN performance problem (e.g. mtu/encryption performance related). We can also exclude bandwidth trottleing by the VPS provider and the ISP. * Did something essential change in `pf`? [4] * Or is the problem related to OpenBSD's Xen drivers? Could someone help us track down the bottleneck? Any help and hints are very much appreciated. Thank you kindly Berry PS: for a better viewing experience you may compile this email body with `asciidoc` == Environment === Server * OpenBSD 6.2 / amd64 (-release) + syspatch * OpenVPN 2.4.4 * On Virtual Private Server / Xen version "4.9.0" by Xen Project [0] * Detected CPU: Intel(R) Xeon(R) CPU E5-2620 * Detected network device: xnf0 * Firewall configuration: /etc/pf.conf [1] * System Message Buffer [2] === Clients * OpenBSD 6.2 with OpenVPN 2.4.4 * GNU/Linux Gentoo with OpenVPN 2.4.4 * LinesageOS 14.1 with OpenVPN for Android 0.6.73 == Detailed Problem Description / Test Results Please note: the following documented tests used one and the same client / network connection: * GNU/Linux Gentoo with OpenVPN 2.4.4 * Connected to router via wifi on internet connection with max 50Mbit/s download To rule out problems with the client local network settings tests with other client setups on other networks were also performed and showed identical results. For brevity they are not documented here. === Case 1: Server <==> WAN (ok) * When on the server, downloading a file from WAN * Scenario: downloaded 100MB file from http://fra36-speedtest-1.tele2.net/ with curl * Average Download Speed: ~ 10Mbit/s * Testresult: $ curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 100M 100 100M0 0 9309k 0 0:00:11 0:00:11 --:--:-- 10.9M === Case 2: Client <= VPN => Server (ok) * When on the client, downloading a file from server via VPN tunnel * Scenario: standard download test with `iperf` * Average Download Speed: ~ 15Mbit/s * Testresult: # iperf -s --- Server listening on TCP port 5001 TCP window size: 16.0 KByte (default) --- [ 4] local 10.8.0.1 port 5001 connected with 10.8.0.4 port 34998 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.2 sec 18.5 MBytes 15.2 Mbits/sec # iperf -c 10.8.0.1 --- Client connecting to 10.8.0.1, TCP port 5001 TCP window size: 45.0 KByte (default) --- [ 3] local 10.8.0.4 port 34998 connected with 10.8.0.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 18.5 MBytes 15.5 Mbits/sec === Case 3a: Client <= VPN => Server <==> WAN (broken) * When on the client, downloading a file from WAN via VPN tunnel * Scenario: downloaded 100MB file from http://fra36-speedtest-1.tele2.net/ with curl * Average Download Speed: ~ 5KB/s * Testresult: curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 100M0 149k0 0 5102 0 5:42:32 0:00:30 5:42:02 4933 === Case 3b: Client <==> WAN (ok) * When on the client, downloading a file from WAN directly * Scenario: downloaded 100MB file from http://fra36-speedtest-1.tele2.net/ with curl * Average Download Speed: ~ 1100KB/s * Testresult: curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 100M 100 100M0 0 1113k 0 0:01:32 0:01:32 --:--:-- 1196k == Previous working system Before the upgrade to OpenBSD 6.2 we had a working system with the following setup: * OpenBSD 6.1 / i386 * OpenVPN 2.4.1 * firewall settings were the same [8] The fact that we had installed i386 instead of amd64 was unintentional. We had to change the virtual machine (QEMU) network interface from Realtek to Virtio to get a good performance on the external network interface. Hence the working system's external interface was operating on `vio`. The following system message buffer still lists the inefficient `re` device. * System Message Buffer [3] == Appendix * [0] http
