Hello Tom
Thank you very much for your hint.
I have disabled pf with "pfctrl -d" command but didn't notice any
difference in the 10GBit transfer speed.
The CPU usage was high (like 100% for one of the available CPU cores)
# Single send
obsdsrv2$ scp 4GByte-random.bin ironm@10.0.0.2:/home/ironm/send4GByte-v1.bin
4GByte-random.bin 89% 3665MB 71.0MB/s 00:06 ETA
---
# Send and receive at once (two scp connections)
obsdsrv2$ scp 4GByte-random.bin ironm@10.0.0.2:/home/ironm/send4GByte-v1.bin
4GByte-random.bin 50% 2050MB 47.7MB/s 00:42 ETA
obsdsrv2$ scp ironm@10.0.0.2:/home/ironm/4GByte-random.bin
receive4GByte-v1.bin
4GByte-random.bin 68% 2814MB 50.8MB/s 00:25 ETA
Details of the test are in the attached file:
obsd2obsd-send_conf_AMD_FX_4100_to_Xeon-E31270-10gbit-NICs-SSD_cpu-load-top.txt
Kind regards
Mark
--
m...@it-infrastrukturen.org
Am 08.04.2019 00:20, schrieb Tom Smyth:
Hello
if you disable pf you should get alot higher speeds,
as PF uses 1 CPU
alternatively you can enable experimental pf code that uses more than one
CPU
On Sun, 7 Apr 2019 at 23:15, Mark Schneider <m...@it-infrastrukturen.org> wrote:
Short feedback:
Just for the test I have checked the 10GBit network performance
between two FreeBSD 13.0 servers (both HP DL380g7 machines)
transfering data in both directions
# ---
ironm@fbsdsrv2:~ $ scp ironm@200.0.0.10:/home/ironm/t2.iso t100.iso
Password for ironm@fbsdsrv1:
t2.iso 100% 3626MB 130.2MB/s 00:27
# ---
ironm@fbsdsrv2:~ $ scp obsd2fbsd.iso ironm@200.0.0.10:/home/ironm/t1.iso
Password for ironm@fbsdsrv1:
obsd2fbsd.iso 100% 3626MB 140.4MB/s 00:25
# ---
The ssh performance using 10GBit network connection on FreeBSD 13.0
is approx 7 times higher than the one on OpenBSD 6.4.
Is it the question of the "ix" NIC driver of OpenBSD 6.4?
(X520-DA2 NICs from Intel)
Does one of you achieve good 10Gbit network performance with other
10Gbit NICs?
Thank you in advance for your hints.
Kind regards
Mark
--
m...@it-infrastrukturen.org
Am 06.04.2019 22:52, schrieb Mark Schneider:
Hi,
Please allow me few questions regarding 10GBit network performance on
OpenBSD 6.4.
I face quite low network performance for the Intell X520-DA2 10GBit
network card.
Test configuration in OpenBSD-Linux-10GBit_net_performance.txt -
http://paste.debian.net/1076461/
Low transfer rate for scp - OpenBSD-10GBit-perftest.txt -
http://paste.debian.net/1076460/
Test configuration:
# ---
# OpenBSD 6.4 on HP DL380g7
# -------------------------
# 10GBit X520-DA2 NIC
ix0: flags=208843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6>
mtu 1500
media: Ethernet autoselect (10GbaseSR
full-duplex,rxpause,txpause)
inet6 fe80::d51e:1b74:17d7:8230%ix0 prefixlen 64 scopeid 0x1
inet 200.0.0.3 netmask 0xffffff00 broadcast 200.0.0.255
ix1: flags=208843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6>
mtu 1500
media: Ethernet autoselect (10GbaseSR
full-duplex,rxpause,txpause)
inet 10.0.0.7 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::b488:caea:5d6f:9992%ix1 prefixlen 64 scopeid 0x2
# ---
Compare to Linux the 10GBit transfer from/to OpenBSD is few times slower:
# ---
# OpenBSD to Linux (Asus P8BWS)
# -----------------------------
srvob# iperf3 -c 10.0.0.2
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.50 GBytes 1.29 Gbits/sec
sender
[ 5] 0.00-10.20 sec 1.50 GBytes 1.27 Gbits/sec
receiver
# ---
# ---
# Linux (DL380g7) to Linux (Asus P8BWS)
# -------------------------------------
root@kali:~# iperf3 -c 100.0.0.2
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 328
sender
[ 5] 0.00-10.04 sec 10.9 GBytes 9.35 Gbits/sec
receiver
# ---
The scp transfer rate is like 21MBytes/s only per ssh connection
(OpenBSD <-> Linux):
# ---
root@kali:~# scp /re*/b*/ka*/kali-linux-kde-2019.1a-*.iso
ironm@10.0.0.7:/home/ironm/t12.iso
ironm@10.0.0.7's password:
kali-linux-kde-2019.1a-amd64.iso 4% 173MB
21.5MB/s 02:40 ETA
# ---
The 1GBit cooper based NIC works also slower but reaching almost 40%
of the max trasfer rate of 1 Gbit:
# ---
# OpenBSD 6.4 (DL380g7 1Gbit NIC) to Linux (DL380g7 1GBit NIC)
# ------------------------------------------------------------
srvob# iperf3 -c 170.0.0.10
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 471 MBytes 395 Mbits/sec
sender
[ 5] 0.00-10.20 sec 471 MBytes 388 Mbits/sec
receiver
# ---
# ---
# Linux (Asus P8BWS) to Linux (DL380g7)
# -------------------------------------
root@kali:~# iperf3 -c 192.168.1.122
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec 183
sender
[ 5] 0.00-10.04 sec 1.09 GBytes 934 Mbits/sec
receiver
# ---
Thank you in advance for your hints what OpenBSD 6.4 settings do I miss.
Best regards
Mark
Xeon E31270 CPU Server
load averages: 0.00, 0.00, 0.03
obsdsrv1.born2b3.net 00:38:06
46 processes: 45 idle, 1 on processor
up 2:08
CPU0 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU1 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU2 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU3 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU4 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU5 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU6 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU7 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
Memory: Real: 69M/886M act/tot Free: 30G Cache: 712M Swap: 0K/32G
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
20673 root 2 0 1092K 3756K idle poll 10:50 0.00% sshd
18870 _slaacd 2 0 732K 788K idle kqread 0:05 0.00% slaacd
36852 _dhcp 2 0 772K 692K idle poll 0:05 0.00% dhclient
51148 root 18 0 964K 992K idle pause 0:04 0.00% ksh
67419 root 28 0 1104K 2964K onproc/0 - 0:01 0.00% top
AMD FX 4100 CPU Server
load averages: 0.06, 0.02, 0.04
obsdsrv2.born2b3.net 00:38:38
54 processes: 53 idle, 1 on processor
up 1:35
CPU0 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU1 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU2 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU3 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
Memory: Real: 109M/873M act/tot Free: 30G Cache: 658M Swap: 0K/32G
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
19929 _x11 2 0 27M 40M sleep/0 poll 0:12 0.00% Xorg
1 root 10 0 404K 456K idle wait 0:01 0.00% init
20145 ironm 2 0 920K 3456K sleep/2 select 0:01 0.00% fvwm
62507 ironm 2 0 7004K 11M sleep/0 select 0:00 0.00% xterm
# ------------------------------------------------------------------------------
# Single send
obsdsrv2$ scp 4GByte-random.bin ironm@10.0.0.2:/home/ironm/send4GByte-v1.bin
4GByte-random.bin 89% 3665MB
71.0MB/s 00:06 ETA
# ---
load averages: 0.30, 0.06, 0.04
obsdsrv1.born2b3.net 00:39:51
50 processes: 48 idle, 2 on processor
up 2:10
CPU0 states: 0.6% user, 0.0% nice, 1.0% sys, 0.2% spin, 3.6% intr, 94.6%
idle
CPU1 states: 20.2% user, 0.0% nice, 28.7% sys, 6.6% spin, 0.0% intr, 44.5%
idle
CPU2 states: 11.8% user, 0.0% nice, 25.7% sys, 4.8% spin, 0.0% intr, 57.7%
idle
CPU3 states: 9.6% user, 0.0% nice, 11.8% sys, 4.0% spin, 0.0% intr, 74.7%
idle
CPU4 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU5 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU6 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU7 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
Memory: Real: 71M/889M act/tot Free: 30G Cache: 712M Swap: 0K/32G
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
27857 ironm 39 0 1220K 3024K onproc/1 - 0:11 37.45% sshd
47712 ironm 2 0 428K 1456K sleep/2 poll 0:02 9.77% scp
20673 root 2 0 1092K 3756K idle poll 10:50 0.00% sshd
18870 _slaacd 2 0 732K 788K idle kqread 0:05 0.00% slaacd
load averages: 0.43, 0.10, 0.07
obsdsrv2.born2b3.net 00:39:58
56 processes: 54 idle, 2 on processor
up 1:37
CPU0 states: 33.7% user, 0.0% nice, 27.1% sys, 10.6% spin, 6.0% intr, 22.6%
idle
CPU1 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU2 states: 31.7% user, 0.0% nice, 34.9% sys, 10.0% spin, 0.0% intr, 23.4%
idle
CPU3 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
Memory: Real: 112M/877M act/tot Free: 30G Cache: 658M Swap: 0K/32G
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
35786 ironm 10 0 3108K 5888K onproc/2 - 0:20 61.82% ssh
21137 ironm 2 0 420K 1496K sleep/2 poll 0:05 13.53% scp
19929 _x11 2 0 27M 40M sleep/2 poll 0:12 1.37% Xorg
# ------------------------------------------------------------------------------
# Send and receive at once (two scp connections)
obsdsrv2$ scp 4GByte-random.bin ironm@10.0.0.2:/home/ironm/send4GByte-v1.bin
ironm@10.0.0.2's password:
4GByte-random.bin 50% 2050MB
47.7MB/s 00:42 ETA
obsdsrv2$ scp ironm@10.0.0.2:/home/ironm/4GByte-random.bin receive4GByte-v1.bin
ironm@10.0.0.2's password:
4GByte-random.bin 68% 2814MB 50.8MB/s
00:25 ETA
load averages: 1.33, 0.40, 0.18
obsdsrv1.born2b3.net 01:06:11
57 processes: 54 idle, 3 on processor
up 2:36
CPU0 states: 11.4% user, 0.0% nice, 15.4% sys, 9.8% spin, 10.0% intr, 53.5%
idle
CPU1 states: 30.1% user, 0.0% nice, 24.8% sys, 14.0% spin, 0.0% intr, 31.1%
idle
CPU2 states: 21.0% user, 0.0% nice, 31.1% sys, 11.8% spin, 0.0% intr, 36.1%
idle
CPU3 states: 21.6% user, 0.0% nice, 23.6% sys, 12.6% spin, 0.0% intr, 42.3%
idle
CPU4 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU5 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU6 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU7 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
Memory: Real: 84M/906M act/tot Free: 30G Cache: 712M Swap: 0K/32G
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
96628 ironm 10 0 1176K 3060K onproc/1 - 0:41 66.46% sshd
32758 ironm 64 0 4892K 6780K onproc/0 - 0:41 61.67% sshd
39950 ironm 2 0 432K 1440K sleep/3 poll 0:08 13.09% scp
90607 ironm 2 0 436K 1480K sleep/3 poll 0:09 12.84% scp
59276 root 2 0 1044K 3764K sleep/2 poll 0:34 0.00% sshd
13599 root 2 0 1044K 3752K sleep/3 poll 0:09 0.00% sshd
18870 _slaacd 2 0 732K 788K sleep/3 kqread 0:05 0.00% slaacd
load averages: 2.44, 0.76, 0.33
obsdsrv2.born2b3.net 01:06:14
60 processes: 3 running, 56 idle, 1 on processor
up 2:03
CPU0 states: 38.3% user, 0.0% nice, 42.7% sys, 13.6% spin, 5.4% intr, 0.0%
idle
CPU1 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
CPU2 states: 40.9% user, 0.0% nice, 44.1% sys, 14.8% spin, 0.0% intr, 0.2%
idle
CPU3 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr, 100%
idle
Memory: Real: 120M/887M act/tot Free: 30G Cache: 658M Swap: 0K/32G
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
12350 ironm 64 0 1156K 3924K run/0 - 0:49 66.21% ssh
78915 ironm 64 0 4560K 7336K run/2 - 0:47 64.16% ssh
11917 ironm 48 0 424K 1512K run/2 - 0:09 13.53% scp
1584 ironm 2 0 440K 1528K sleep/2 poll 0:08 9.96% scp
19929 _x11 2 0 27M 40M sleep/2 poll 0:14 0.98% Xorg
1 root 10 0 404K 456K idle wait 0:01 0.00% init
# ---
obsdsrv2$ scp 4GByte-random.bin ironm@10.0.0.2:/home/ironm/send4GByte-v1.bin
4GByte-random.bin 100% 4096MB 49.4MB/s 01:22
obsdsrv2$ scp ironm@10.0.0.2:/home/ironm/4GByte-random.bin receive4GByte-v1.bin
4GByte-random.bin 100% 4096MB 51.2MB/s 01:19
