Re: Coming from FreeBSD, lower networking performance
Funny, cause I have almost identical speeds to/from my OpenBSD to/from my Debian, but different results between OpenBSD to OpenBSD and Debian to Debian. All servers are sitting on a 4x1GB LACP-link in the same network switch *OpenBSD 6.3 to OpenBSD 6.3* **# iperf3 -4 -c ns1.semarkit.net Connecting to host ns1.semarkit.net, port 5201 [ 4] local 192.168.245.1 port 47049 connected to 192.168.245.9 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 56.3 MBytes 472 Mbits/sec [ 4] 1.00-2.00 sec 64.0 MBytes 537 Mbits/sec [ 4] 2.00-3.00 sec 64.8 MBytes 543 Mbits/sec [ 4] 3.00-4.00 sec 66.0 MBytes 554 Mbits/sec [ 4] 4.00-5.00 sec 64.8 MBytes 544 Mbits/sec [ 4] 5.00-6.00 sec 67.1 MBytes 562 Mbits/sec [ 4] 6.00-7.00 sec 66.7 MBytes 558 Mbits/sec [ 4] 7.00-8.00 sec 63.3 MBytes 533 Mbits/sec [ 4] 8.00-9.00 sec 65.2 MBytes 547 Mbits/sec [ 4] 9.00-10.00 sec 66.5 MBytes 558 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 645 MBytes 541 Mbits/sec sender [ 4] 0.00-10.00 sec 645 MBytes 541 Mbits/sec receiver iperf Done. *OpenBSD 6.3 to Debian Stretch* # iperf3 -4 -c 192.168.245.7 Accepted connection from 192.168.245.7, port 55458 [ 5] local 192.168.245.9 port 5201 connected to 192.168.245.7 port 55460 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 96.0 MBytes 805 Mbits/sec [ 5] 1.00-2.00 sec 136 MBytes 1.14 Gbits/sec [ 5] 2.00-3.00 sec 148 MBytes 1.25 Gbits/sec [ 5] 3.00-4.00 sec 163 MBytes 1.37 Gbits/sec [ 5] 4.00-5.00 sec 179 MBytes 1.50 Gbits/sec [ 5] 5.00-6.00 sec 178 MBytes 1.49 Gbits/sec [ 5] 6.00-7.00 sec 185 MBytes 1.55 Gbits/sec [ 5] 7.00-8.00 sec 179 MBytes 1.50 Gbits/sec [ 5] 8.00-9.00 sec 183 MBytes 1.54 Gbits/sec [ 5] 9.00-10.00 sec 183 MBytes 1.53 Gbits/sec [ 5] 10.00-10.00 sec 0.00 Bytes 0.00 bits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.00 sec 1.59 GBytes 1.37 Gbits/sec receiver --- *Debian Stretch to OpenBSD 6.3 * # iperf3 -4 -c ns1.semarkit.net Connecting to host ns1.semarkit.net, port 5201 [ 4] local 192.168.245.7 port 55460 connected to 192.168.245.9 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 0 55.1 KBytes [ 4] 1.00-2.00 sec 136 MBytes 1.14 Gbits/sec 0 90.5 KBytes [ 4] 2.00-3.00 sec 149 MBytes 1.25 Gbits/sec 0 124 KBytes [ 4] 3.00-4.00 sec 163 MBytes 1.37 Gbits/sec 0 157 KBytes [ 4] 4.00-5.00 sec 179 MBytes 1.50 Gbits/sec 0 191 KBytes [ 4] 5.00-6.00 sec 178 MBytes 1.49 Gbits/sec 0 225 KBytes [ 4] 6.00-7.00 sec 185 MBytes 1.56 Gbits/sec 0 266 KBytes [ 4] 7.00-8.00 sec 179 MBytes 1.51 Gbits/sec 0 296 KBytes [ 4] 8.00-9.00 sec 184 MBytes 1.54 Gbits/sec 0 327 KBytes [ 4] 9.00-10.00 sec 183 MBytes 1.54 Gbits/sec 0 375 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.59 GBytes 1.37 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 1.59 GBytes 1.37 Gbits/sec receiver iperf Done. *Debian Stretch to Debian Stretch* # iperf3 -4 -c 192.168.245.231 Connecting to host 192.168.245.231, port 5201 [ 4] local 192.168.245.7 port 55028 connected to 192.168.245.231 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 269 MBytes 2.26 Gbits/sec 0 1.72 MBytes [ 4] 1.00-2.00 sec 265 MBytes 2.22 Gbits/sec 0 2.17 MBytes [ 4] 2.00-3.00 sec 232 MBytes 1.95 Gbits/sec 494 1.13 MBytes [ 4] 3.00-4.00 sec 254 MBytes 2.13 Gbits/sec 0 1.17 MBytes [ 4] 4.00-5.00 sec 291 MBytes 2.44 Gbits/sec 0 1.17 MBytes [ 4] 5.00-6.00 sec 287 MBytes 2.40 Gbits/sec 0 1.17 MBytes [ 4] 6.00-7.00 sec 297 MBytes 2.49 Gbits/sec 0 1.17 MBytes [ 4] 7.00-8.00 sec 289 MBytes 2.42 Gbits/sec 0 1.17 MBytes [ 4] 8.00-9.00 sec 301 MBytes 2.53 Gbits/sec 0 1.17 M
Re: Coming from FreeBSD, lower networking performance
On 2018-07-13, Sijmen J. Mulder wrote: > After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking > performance. Yes, that's not a huge surprise, FreeBSD has taken speed as quite a high priority, OpenBSD has concentrated more on other areas. As more of the kernel becomes parallelised between multiple cpu cores and bottlenecks are identified and fixed, this should improve.
Re: Coming from FreeBSD, lower networking performance
Hi, On ven. 13 juil. 21:46:24 2018, bit shifter wrote: > You could try running iperf to eliminate disk IO from the equation and > narrow down the potential sources of the performance deficit you're > seeing. By curiosity, I ran a test here, on a already production running infra. FreeBSD 11.1 is 20 times faster than OpenBSD 6.3; 10Gbps vs. 500M. My raw terminal output is here: https://paste.swordarmor.fr/raw/vlix -- alarig
Re: Coming from FreeBSD, lower networking performance
On 14.7.2018 01:20, Sijmen J. Mulder wrote: Hi all, After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance. Both incoming and outgoing traffic seems to be 2-3 times slower on average. By testing 100MB file transfers I've mostly eliminated the following factors: As well "define" slow. $ time curl -O http://ftp.fi.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/xsrc.tar.gz % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 123M 100 123M0 0 7646k 0 0:00:16 0:00:16 --:--:-- 9242k 0m16.58s real 0m00.09s user 0m03.24s system $ That is OpenBSD 64bit snapshot, running as guest in VirtualBox on Windows 10, 2 years old laptop, connected via WiFi in crowded space (at least 15 other networks around) to not-perfect Euro DOCSIS 3.0 CPE via provider structure and from Czech Republic to Finland mirror of NetBSD Theoretical maximum on my CPE is 18,75MB/s so having those speeds above is pretty nice in such environment. BTW just for fun let's see how Windows 10 host deal with that. λ curl -O http://ftp.fi.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/xsrc.tar.gz % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 123M 100 123M0 0 661k 0 0:03:11 0:03:11 --:--:-- 704k WHOAAA :-) - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS) - Client software - Peer host - VM provider/platform The easiest way to show and test this difference is as such: 1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM 2. Install curl, then `time curl -O http://download.thinkbroadband.com/100MB.zip` On both my VPS provider and my own PC with VirtualBox VMs the difference is about 3x for the above test. Similar results happen when scp-ing a file to the machine. Any ideas on the cause of this? Any additional tests or tweaks I could try? Sijmen
Re: Coming from FreeBSD, lower networking performance
On 14.7.2018 01:20, Sijmen J. Mulder wrote: Hi all, After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance. Both incoming and outgoing traffic seems to be 2-3 times slower on average. By testing 100MB file transfers I've mostly eliminated the following factors: - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS) - Client software - Peer host - VM provider/platform The easiest way to show and test this difference is as such: 1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM 2. Install curl, then `time curl -O http://download.thinkbroadband.com/100MB.zip` 1. What machine HW and VM are you testing? dmesg? 2. That page/file does not exist, but is that delay consistent with eg. ftp(1) use instead of curl? On both my VPS provider and my own PC with VirtualBox VMs the difference is about 3x for the above test. Similar results happen when scp-ing a file to the machine. VirtualBox is real crap when it comes to xBSD (not that it's much better for others anyway). No matter what you set especially I/O will be terrible Any ideas on the cause of this? Any additional tests or tweaks I could try? systat, vmstat, top... to find where is it showing bottleneck As well do not tweak anything unless you can be sure what is the weakpoint. And then you may find that there is however nothing to tweak. Most easy tweak for start is to install -current and test there Sijmen
Re: Coming from FreeBSD, lower networking performance
Hello Sijmen, On 07/13/18 16:20, Sijmen J. Mulder wrote: Hi all, After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance. Both incoming and outgoing traffic seems to be 2-3 times slower on average. By testing 100MB file transfers I've mostly eliminated the following factors: The easiest way to show and test this difference is as such: 1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM 2. Install curl, then `time curl -O http://download.thinkbroadband.com/100MB.zip` On both my VPS provider and my own PC with VirtualBox VMs the difference is about 3x for the above test. Similar results happen when scp-ing a file to the machine. Please do some benchmarks using -current and send in the results. The OpenBSD devs don't have time to do benchmarking, and always appreciate proper benchmarking comparisons. For your networking tests, please don't do any tests that involve disk writes. Disk speed is where you may have issues, as most disk operations/drivers are not multithreaded and/or kernel biglocked still. For example, when using identical hardware (an old core 2 duo Dell workstation), I can read from my Samsung 860 SSD at ~325MB/s on CentOS. On OpenBSD, I get more like ~130MB/s. if you want to test ssh/scp speed, make sure you write/output all ssh/scp data transfers to /dev/null. What sort of data pipe are you working with? I have found myself able to saturate a gigabit line fairly easily over ssh using modest hardware. Please reply back with some proper benchmarks as I'm sure many of us would love to take a look. Cheers, Jordan
Re: Coming from FreeBSD, lower networking performance
Hello Sijmen, this is not something I remember seeing often on the list; to improve your chances of replies beyond "use iperf to test" (there's tcpbench(1) in base) you should at least provide a dmesg(8). Maybe even use sendbug(1) and have the report go to bugs@ [1]. Could you run -current [2] to see if the problem is still there? [1] http://www.openbsd.org/report.html [2] http://www.openbsd.org/faq/current.html Marcus i...@sjmulder.nl (Sijmen J. Mulder), 2018.07.14 (Sat) 01:20 (CEST): > Hi all, > > After migrating a VPS from FreeBSD to OpenBSD I noticed reduced > networking performance. Both incoming and outgoing traffic seems to be > 2-3 times slower on average. By testing 100MB file transfers I've > mostly eliminated the following factors: > > - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS) > - Client software > - Peer host > - VM provider/platform > > The easiest way to show and test this difference is as such: > > 1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM > 2. Install curl, then `time curl -O > http://download.thinkbroadband.com/100MB.zip` > > On both my VPS provider and my own PC with VirtualBox VMs the > difference is about 3x for the above test. Similar results happen when > scp-ing a file to the machine. > > Any ideas on the cause of this? Any additional tests or tweaks I could > try? > > Sijmen >
Re: Coming from FreeBSD, lower networking performance
Have you looked into IPerf? https://iperf.fr/ This is what I typically use for testing network throughput. Downloading a file is a bit more complex and involves things like the source server/latency/etc. as well as disk performance. (I know a 100MB file isnt much but still...) IPerf has a lot of knobs to tweak, like number of threads, TCP window size, etc. You will need 2 hosts, one to act as server and one as client. I recommend the latest stable version of iperf3, which is available as a package: $ doas pkg_add iperf3 Hope this helps. On Fri, Jul 13, 2018 at 7:20 PM, Sijmen J. Mulder wrote: > Hi all, > > After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking > performance. Both incoming and outgoing traffic seems to be 2-3 times > slower on average. By testing 100MB file transfers I've mostly eliminated > the following factors: > > - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS) > - Client software > - Peer host > - VM provider/platform > > The easiest way to show and test this difference is as such: > > 1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM > 2. Install curl, then `time curl -O http://download. > thinkbroadband.com/100MB.zip` > > On both my VPS provider and my own PC with VirtualBox VMs the difference > is about 3x for the above test. Similar results happen when scp-ing a file > to the machine. > > Any ideas on the cause of this? Any additional tests or tweaks I could try? > > Sijmen > >
Re: Coming from FreeBSD, lower networking performance
On Fri, Jul 13, 2018 at 5:20 PM, Sijmen J. Mulder wrote: > Any ideas on the cause of this? Any additional tests or tweaks I could try? You could try running iperf to eliminate disk IO from the equation and narrow down the potential sources of the performance deficit you're seeing.
Coming from FreeBSD, lower networking performance
Hi all, After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance. Both incoming and outgoing traffic seems to be 2-3 times slower on average. By testing 100MB file transfers I've mostly eliminated the following factors: - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS) - Client software - Peer host - VM provider/platform The easiest way to show and test this difference is as such: 1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM 2. Install curl, then `time curl -O http://download.thinkbroadband.com/100MB.zip` On both my VPS provider and my own PC with VirtualBox VMs the difference is about 3x for the above test. Similar results happen when scp-ing a file to the machine. Any ideas on the cause of this? Any additional tests or tweaks I could try? Sijmen
