Re: IPSec heavy traffic slows down all network traffic
Hello, i replaced the MP kernel with the SP one and made some tests. Perfomances are better, all cpu goes to the kernel and user processes. But it is slow. I will ask to change the hardware, as it is old. jy boisiaud Le mer. 22 juil. 2020 à 08:36, jean-yves boisiaud < jean-yves.boisi...@alcor-consulting.fr> a écrit : > ok, i'll try with the bsd.sp kernel. > > thank you for your help. > > :-( > > > Le dim. 19 juil. 2020 à 07:41, Chris Cappuccio a > écrit : > >> jean-yves boisiaud [jean-yves.boisi...@alcor-consulting.fr] wrote: >> > Last week, I upgraded a couple of firewalls using carp/pfsync and >> sasyncd >> > from 6.0 to 6.7 (yes, big jump !). >> > >> > I also applied all the 6.7 published patches. >> > >> > When some heavy traffic takes one of the IPSec tunnel, I noticed that : >> > - all network connections are slowed down >> > - unused network bandwidth increase instead of decrease >> > - idle CPU move towards 0, and spinning increase to take about 50% of >> the >> > CPU >> > >> > When I stop the IPSec traffic : >> > - network connections increase immediatly >> > - unused network bandwidth cecreases immediately >> > - spinning CPU is low. >> > >> >> This is basically a performance regression that could be due to the MP >> work. You are seemingly running into contention that wasn't possible >> before. >> The question is, where is this happening? I don't know if the dynamic >> tracer >> can help here. >> > > > -- > Jean-Yves Boisiaud - Alcor Consulting > 49, rue du Chemin Vert > 49300 Cholet > mobile : +33 6 63 71 73 46 >
Re: IPSec heavy traffic slows down all network traffic
ok, i'll try with the bsd.sp kernel. thank you for your help. :-( Le dim. 19 juil. 2020 à 07:41, Chris Cappuccio a écrit : > jean-yves boisiaud [jean-yves.boisi...@alcor-consulting.fr] wrote: > > Last week, I upgraded a couple of firewalls using carp/pfsync and sasyncd > > from 6.0 to 6.7 (yes, big jump !). > > > > I also applied all the 6.7 published patches. > > > > When some heavy traffic takes one of the IPSec tunnel, I noticed that : > > - all network connections are slowed down > > - unused network bandwidth increase instead of decrease > > - idle CPU move towards 0, and spinning increase to take about 50% of the > > CPU > > > > When I stop the IPSec traffic : > > - network connections increase immediatly > > - unused network bandwidth cecreases immediately > > - spinning CPU is low. > > > > This is basically a performance regression that could be due to the MP > work. You are seemingly running into contention that wasn't possible > before. > The question is, where is this happening? I don't know if the dynamic > tracer > can help here. > -- Jean-Yves Boisiaud - Alcor Consulting 49, rue du Chemin Vert 49300 Cholet mobile : +33 6 63 71 73 46
Re: IPSec heavy traffic slows down all network traffic
jean-yves boisiaud [jean-yves.boisi...@alcor-consulting.fr] wrote: > Last week, I upgraded a couple of firewalls using carp/pfsync and sasyncd > from 6.0 to 6.7 (yes, big jump !). > > I also applied all the 6.7 published patches. > > When some heavy traffic takes one of the IPSec tunnel, I noticed that : > - all network connections are slowed down > - unused network bandwidth increase instead of decrease > - idle CPU move towards 0, and spinning increase to take about 50% of the > CPU > > When I stop the IPSec traffic : > - network connections increase immediatly > - unused network bandwidth cecreases immediately > - spinning CPU is low. > This is basically a performance regression that could be due to the MP work. You are seemingly running into contention that wasn't possible before. The question is, where is this happening? I don't know if the dynamic tracer can help here.
Re: IPSec heavy traffic slows down all network traffic
On 17.7.2020. 20:17, jean-yves boisiaud wrote: > hello, > > Last week, I upgraded a couple of firewalls using carp/pfsync and sasyncd > from 6.0 to 6.7 (yes, big jump !). > > I also applied all the 6.7 published patches. > > When some heavy traffic takes one of the IPSec tunnel, I noticed that : > - all network connections are slowed down > - unused network bandwidth increase instead of decrease > - idle CPU move towards 0, and spinning increase to take about 50% of the > CPU > > When I stop the IPSec traffic : > - network connections increase immediatly > - unused network bandwidth cecreases immediately > - spinning CPU is low. > > Yes I know, my hardware is a bit old. I understand that CPU raises due to > IPSec crypto, but I do not understand why network performance decrease. maybe intel mitigation stuff decreased your performance. it in from openbsd 6.3 ... don't know if you are using aes for ipsec, but you cpu doesn't have aes-ni... maybe to try wireguard ? :)
IPSec heavy traffic slows down all network traffic
hello, Last week, I upgraded a couple of firewalls using carp/pfsync and sasyncd from 6.0 to 6.7 (yes, big jump !). I also applied all the 6.7 published patches. When some heavy traffic takes one of the IPSec tunnel, I noticed that : - all network connections are slowed down - unused network bandwidth increase instead of decrease - idle CPU move towards 0, and spinning increase to take about 50% of the CPU When I stop the IPSec traffic : - network connections increase immediatly - unused network bandwidth cecreases immediately - spinning CPU is low. Yes I know, my hardware is a bit old. I understand that CPU raises due to IPSec crypto, but I do not understand why network performance decrease. 1) Situation before doing anything: # pktstat -ntT -m 1 -i em1 interface: em1total: 122.6Mb (7m18s) cur: 260.1k (0%) min: 0.0 max: 100.0M avg: 279.3k bps bps% b desc 69.6k 0% 348.6k tcp 109.7.96.229:54880 <-> 52.113.194.132:443 60.0k 0% 36.1M ip proto 50 109.7.96.226 <-> 92.174.146.73 36.5k 0% 182.8k tcp 109.7.96.229:59950 <-> 52.113.194.132:443 12.3k 0% 61.5k tcp 109.7.96.229:51009 <-> 216.58.214.78:443 11.8k 0% 58.9k tcp 109.7.96.229:61287 <-> 216.58.206.229:443 # top load averages: 0.14, 0.12, 0.14 ..fr 20:00:05 81 processes: 2 running, 77 idle, 2 on processor up 10:53 CPU0: 31.9% user, 0.0% nice, 21.4% sys, 5.8% spin, 0.4% intr, 40.5% idle CPU1: 30.9% user, 0.0% nice, 17.2% sys, 5.2% spin, 0.0% intr, 46.7% idle Memory: Real: 166M/403M act/tot Free: 561M Cache: 128M Swap: 0K/0K PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND 35828 osadmin 520 1676K 3504K run/0 - 0:03 8.35% sshd 68723 _openvpn 20 4016K 6404K sleep/1 poll 11:41 1.12% openvpn 16143 root 20 1372K 4056K sleep/0 poll 0:00 0.49% sshd 95804 root 280 5440K 6892K run/0 - 0:05 0.34% pktstat 2) Making heavy traffic NOT using IPSec : Notice bandwidth usage. heavy traffic NOT using the IPSec tunnel # ssh ardee dd if=/dev/urandom bs=1M | dd of=/dev/null bs=1M 0+12031 records in 0+12031 records out 198180864 bytes (198 MB, 189 MiB) copied, 23.3799 s, 8.5 MB/s 0+19257 records in 0+19257 records out 316571648 bytes (317 MB, 302 MiB) copied, 37.167 s, 8.5 MB/s # pktstat -ntT -m 1 -i em1 interface: em1total: 8.2Gb (11m49s) cur: 72.6M (72%) min: 0.0 max: 100.0M avg: 11.5M bps bps% b desc 72.4M 72% 8.0G tcp 109.7.96.226:63663 <-> 212.83.131.76:2 66.4k 0% 60.2M ip proto 50 109.7.96.226 <-> 92.174.146.73 33.5k 0% 167.7k tcp 109.7.96.229:52670 <-> 52.97.168.210:443 10.3k 0% 7.5M ip proto 112 109.7.96.227 <-> 224.0.0.18 9.2k 0% 46.3k tcp 109.7.96.229:56973 <-> 40.101.92.178:443 # top load averages: 1.11, 0.61, 0.34 billy.basystemes.fr 20:04:41 76 processes: 75 idle, 1 on processor up 10:58 CPU0: 13.8% user, 0.0% nice, 18.6% sys, 1.2% spin, 11.2% intr, 55.3% idle CPU1: 10.2% user, 0.0% nice, 29.3% sys, 0.6% spin, 0.0% intr, 59.9% idle Memory: Real: 166M/390M act/tot Free: 574M Cache: 115M Swap: 0K/0K PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND 95804 root 20 9760K 8696K sleep/1 poll 0:36 15.77% pktstat 68723 _openvpn 20 4012K 6332K sleep/1 poll 11:46 1.17% openvpn 33560 _isakmpd 20 11M 15M sleep/0 select7:28 0.59% isakmpd 83650 _openvpn 20 3928K 6388K sleep/0 poll 20:10 0.00% openvpn 3) Making heavy traffic using the IPSec tunnel in addition to the previous heavy traffic : Notice bandwidth usage, which has decreased, and spinning value in top. Also notice the weak rate tranfer in the IPSec tunnel. heavy traffic NOT using the IPSec tunnel # ssh ardee dd if=/dev/urandom bs=1M | dd of=/dev/null bs=1M 0+11902 records in 0+11902 records out 231751680 bytes (232 MB, 221 MiB) copied, 109.809 s, 2.1 MB/s 0+12372 records in 0+12372 records out 247152640 bytes (247 MB, 236 MiB) copied, 131.151 s, 1.9 MB/s heavy traffic using the IPSec tunnel # ssh doon dd if=/dev/urandom bs=1M | dd of=/tmp/null bs=1M 0+2496 records in 0+2496 records out 81723392 bytes (82 MB, 78 MiB) copied, 91.6991 s, 891 kB/s 0+3078 records in 0+3078 records out 100794368 bytes (101 MB, 96 MiB) copied, 113.042 s, 892 kB/s # pktstat -ntT -m 1 -i em1 interface: em1total: 15.3Gb (13m44s) cur: 11.1M (11%) min: 0.0 max: 100.0M avg: 18.5M bps bps% b desc 6.2M 6% 163.3M ip proto 50 109.7.96.226 <-> 92.174.146.73 4.7M 4% 1.2G tcp 109.7.96.226:52734 <-> 212.83.131.76:2 33.7k 0% 474.5k ip fragments 25.8k 0% 2.5M udp 109.7.96.228:1195 <-> 92.135.30.8:52978 18.2k 0% 9.8M udp 109.7.96.228:1195 <-> 91.166.166.68:17587 17.6k 0% 88.3k tcp 109.7.96.229:443 <-> 213.32.72.115:47700 # top load averages: 2.59, 1.39, 0.70 billy.basystemes.fr 20:08:22
