In my opinion your net5501’s system calls per interval are relatively high.
The (traps sys) column on my firewall hovers between 40 & 50 quite consistently.
My understanding is that system calls are things like program calls & library
access.
In addition your net5501’s memory requests per second seem heavy.
You have fifty eight million 1024 bucket requests per second.
My firewall has a max of one hundred thousand 128 bucket requests per second.
Many commercial routers run a customized kernel & rely on a striped down
user-land.
The kernel is also recompiled to run TCP/IP4 only & can no longer execute
things like storage or virtualization.
The OpenBSD O.S includes all the user-land tools such as ping & top in addition
to a standardized precompiled kernel.
Regards
Patrick
.
>
>
> On Thu, 22 Aug 2019 19:12:55 -0500
> Patrick Dohman <dohmanpatr...@gmail.com> wrote:
>
>> Radek
>>
>> I’ve found that fast networking is actually CPU & memory intensive.
>> Pentium 4 and Xeon's are increasingly a necessity for stable firewalls in my
>> opinion.
>> Keep in mind OpenBSD is a monolithic kernel & isn’t a one to one ratio with
>> a commercial router.
>>
>> What are your context switches & interrupts doing while the VPN is up &
>> traffic is flowing?
>>
>> vmstat -w 4
>>
>> What is your memory high water mark during a peak traffic?
>>
>> vmstat -m
>>
>> Regards
>> Patrick
>>
>>> On Aug 21, 2019, at 12:34 AM, radek <r...@int.pl> wrote:
>>>
>>> Hello Patrick,
>>> I am sorry for the late reply.
>>>
>>>> Do you consider memory an issue?
>>> No, I do not. I have a bunch of old Soekris/net5501-70 and ALIX2d2/2d3,
>>> that I use for VPN testing.
>>> Current testing set (6.5/i386) is net5501-70 <-> ALIX2d3
>>> Production set (6.3/i386) is net5501-70 <-> ALIX2d2
>>> Also have tried net5501-70 <-> net5501-70 - the same VPN problem occurs
>>> It is unlikely that every box has any hardware issue.
>>>
>>>> Unix load average can occasionally be deceiving.
>>> I did not know.
>>>
>>> #### net5501-70 ####
>>> $top -d1 | head -n 4
>>> load averages: 0.05, 0.01, 0.00 RAC-fw65-test.PRAC 10:58:14
>>> 38 processes: 1 running, 35 idle, 1 dead, 1 on processor up 3 days, 18:02
>>> CPU states: 0.5% user, 0.0% nice, 0.4% sys, 0.0% spin, 0.2% intr,
>>> 98.8% idle
>>> Memory: Real: 18M/267M act/tot Free: 222M Cache: 97M Swap: 0K/256M
>>>
>>> #### ALIX2d3 ####
>>> $top -d1 | head -n 4
>>> load averages: 0.00, 0.00, 0.00 mon65.home 07:30:05
>>> 37 processes: 1 running, 35 idle, 1 on processor up 13:46
>>> CPU states: 0.3% user, 0.0% nice, 1.1% sys, 0.0% spin, 0.4% intr,
>>> 98.3% idle
>>> Memory: Real: 125M/223M act/tot Free: 14M Cache: 47M Swap: 73M/256M
>>>
>>>
>>>
>>>> What is the speed of your memory?
>>>> What make of Ethernets are you running?
>>> Dmesgs below
>>>
>>> #### net5501-70 ####
>>> OpenBSD 6.5 (GENERIC) #2: Tue Jul 23 23:08:46 CEST 2019
>>> r...@syspatch-65-i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
>>> real mem = 536363008 (511MB)
>>> avail mem = 511311872 (487MB)
>>> mpath0 at root
>>> scsibus0 at mpath0: 256 targets
>>> mainbus0 at root
>>> bios0 at mainbus0: date 20/80/26, BIOS32 rev. 0 @ 0xfac40
>>> pcibios0 at bios0: rev 2.0 @ 0xf0000/0x10000
>>> pcibios0: pcibios_get_intr_routing - function not supported
>>> pcibios0: PCI IRQ Routing information unavailable.
>>> pcibios0: PCI bus #0 is the last bus
>>> bios0: ROM list: 0xc8000/0xa800
>>> cpu0 at mainbus0: (uniprocessor)
>>> cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD" 586-class)
>>> 500 MHz, 05-0a-02
>>> cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
>>> mtrr: K6-family MTRR support (2 registers)
>>> amdmsr0 at mainbus0
>>> pci0 at mainbus0 bus 0: configuration mode 1 (bios)
>>> 0:20:0: io address conflict 0x6100/0x100
>>> 0:20:0: io address conflict 0x6200/0x200
>>> pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x33
>>> glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES
>>> vr0 at pci0 dev 6 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11,
>>> address 00:00:24:cb:4f:cc
>>> ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
>>> 0x004063, model 0x0034
>>> vr1 at pci0 dev 7 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 5,
>>> address 00:00:24:cb:4f:cd
>>> ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
>>> 0x004063, model 0x0034
>>> vr2 at pci0 dev 8 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 9,
>>> address 00:00:24:cb:4f:ce
>>> ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
>>> 0x004063, model 0x0034
>>> vr3 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 12,
>>> address 00:00:24:cb:4f:cf
>>> ukphy3 at vr3 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
>>> 0x004063, model 0x0034
>>> glxpcib0 at pci0 dev 20 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, 32-bit
>>> 3579545Hz timer, watchdog, gpio, i2c
>>> gpio0 at glxpcib0: 32 pins
>>> iic0 at glxpcib0
>>> pciide0 at pci0 dev 20 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel 0
>>> wired to compatibility, channel 1 wired to compatibility
>>> wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFH-008G>
>>> wd0: 1-sector PIO, LBA48, 7629MB, 15625216 sectors
>>> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
>>> pciide0: channel 1 ignored (disabled)
>>> ohci0 at pci0 dev 21 function 0 "AMD CS5536 USB" rev 0x02: irq 15, version
>>> 1.0, legacy support
>>> ehci0 at pci0 dev 21 function 1 "AMD CS5536 USB" rev 0x02: irq 15
>>> usb0 at ehci0: USB revision 2.0
>>> uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00
>>> addr 1
>>> isa0 at glxpcib0
>>> isadma0 at isa0
>>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>>> com0: console
>>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>>> pckbc0: unable to establish interrupt for irq 12
>>> pckbd0 at pckbc0 (kbd slot)
>>> wskbd0 at pckbd0: console keyboard
>>> pcppi0 at isa0 port 0x61
>>> spkr0 at pcppi0
>>> nsclpcsio0 at isa0 port 0x2e/2: NSC PC87366 rev 9: GPIO VLM TMS
>>> gpio1 at nsclpcsio0: 29 pins
>>> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
>>> usb1 at ohci0: USB revision 1.0
>>> uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00
>>> addr 1
>>> vscsi0 at root
>>> scsibus1 at vscsi0: 256 targets
>>> softraid0 at root
>>> scsibus2 at softraid0: 256 targets
>>> root on wd0a (2bf8b7abbbce37df.a) swap on wd0b dump on wd0b
>>>
>>>
>>> #### ALIX2d3 ####
>>> OpenBSD 6.5 (GENERIC) #2: Tue Jul 23 23:08:46 CEST 2019
>>> r...@syspatch-65-i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
>>> real mem = 267931648 (255MB)
>>> avail mem = 247779328 (236MB)
>>> mpath0 at root
>>> scsibus0 at mpath0: 256 targets
>>> mainbus0 at root
>>> bios0 at mainbus0: date 11/05/08, BIOS32 rev. 0 @ 0xfd088
>>> pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
>>> pcibios0: pcibios_get_intr_routing - function not supported
>>> pcibios0: PCI IRQ Routing information unavailable.
>>> pcibios0: PCI bus #0 is the last bus
>>> bios0: ROM list: 0xe0000/0xa800
>>> cpu0 at mainbus0: (uniprocessor)
>>> cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD" 586-class)
>>> 499 MHz, 05-0a-02
>>> cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
>>> mtrr: K6-family MTRR support (2 registers)
>>> pci0 at mainbus0 bus 0: configuration mode 1 (bios)
>>> pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x33
>>> glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES
>>> vr0 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 10,
>>> address 00:0d:b9:1e:85:8c
>>> ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
>>> 0x004063, model 0x0034
>>> vr1 at pci0 dev 10 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11,
>>> address 00:0d:b9:1e:85:8d
>>> ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
>>> 0x004063, model 0x0034
>>> vr2 at pci0 dev 11 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 15,
>>> address 00:0d:b9:1e:85:8e
>>> ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
>>> 0x004063, model 0x0034
>>> glxpcib0 at pci0 dev 15 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, 32-bit
>>> 3579545Hz timer, watchdog, gpio, i2c
>>> gpio0 at glxpcib0: 32 pins
>>> iic0 at glxpcib0
>>> maxtmp0 at iic0 addr 0x4c: lm86
>>> pciide0 at pci0 dev 15 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel 0
>>> wired to compatibility, channel 1 wired to compatibility
>>> wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFH-008G>
>>> wd0: 1-sector PIO, LBA48, 7629MB, 15625216 sectors
>>> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
>>> pciide0: channel 1 ignored (disabled)
>>> ohci0 at pci0 dev 15 function 4 "AMD CS5536 USB" rev 0x02: irq 12, version
>>> 1.0, legacy support
>>> ehci0 at pci0 dev 15 function 5 "AMD CS5536 USB" rev 0x02: irq 12
>>> usb0 at ehci0: USB revision 2.0
>>> uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00
>>> addr 1
>>> isa0 at glxpcib0
>>> isadma0 at isa0
>>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>>> com0: console
>>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>>> pcppi0 at isa0 port 0x61
>>> spkr0 at pcppi0
>>> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
>>> usb1 at ohci0: USB revision 1.0
>>> uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00
>>> addr 1
>>> nvram: invalid checksum
>>> vscsi0 at root
>>> scsibus1 at vscsi0: 256 targets
>>> softraid0 at root
>>> scsibus2 at softraid0: 256 targets
>>> root on wd0a (83b335c3c86bb80c.a) swap on wd0b dump on wd0b
>>> clock: unknown CMOS layout
>>>
>>> On Mon, 19 Aug 2019 18:17:48 -0500
>>> Patrick Dohman <dohmanpatr...@gmail.com> wrote:
>>>
>>>> Do you consider memory an issue?
>>>> What is the speed of your memory?
>>>> Unix load average can occasionally be deceiving.
>>>> What make of Ethernets are you running?
>>>> Regards
>>>> Patrick
>>>>
>>>>> On Aug 19, 2019, at 5:28 AM, radek <r...@int.pl> wrote:
>>>>>
>>>>> Hello Patrick,
>>>>>
>>>>>> Does your ISP implement authoritative DNS?
>>>>>> Do you suspect a UDP issue?
>>>>> My VPN is configured with IPs, not with domain names. Does DNS and/or UDP
>>>>> matter anyway?
>>>>>
>>>>>> Is a managed (switch) involved?
>>>>> No, it is not. I do not use any switches in my testing setup.
>>>>> GW1--ISP1_modem--.....--ISP2_modem--GW2
>>>>>
>>>>> Has duplex ever been an issue?
>>>>> I have never noticed any duplex issue.
>>>>>
>>>>>
>>>>> On Sun, 18 Aug 2019 16:07:14 -0500
>>>>> Patrick Dohman <dohmanpatr...@gmail.com> wrote:
>>>>>
>>>>>> Does your ISP implement authoritative DNS?
>>>>>> Do you suspect a UDP issue?
>>>>>> Is a managed (switch) involved? Has duplex ever been an issue?
>>>>>> Regards
>>>>>> Patrick
>>>>>>
>>>>>>> On Aug 18, 2019, at 1:03 PM, Radek <r...@int.pl> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have two testing gateways (6.5/i386) with site-to-side VPN between
>>>>>>> its LANs (OpenIKED).
>>>>>>> Both gws are fully syspatched, have public IPs and the same iked/pf
>>>>>>> configuration.
>>>>>>>
>>>>>>> Unfortunately, the network traffic over the VPN tunnel stalls few times
>>>>>>> a day.
>>>>>>>
>>>>>>> On the one side I use a script to monitor VPN tunnel with ping, it
>>>>>>> restarts iked and emails me if there is no ping over the VPN tunnel.
>>>>>>> Date: Sat, 17 Aug 2019 22:10:30 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 06:00:20 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 11:09:00 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 19:03:02 +0200 (CEST)
>>>>>>>
>>>>>>>
>>>>>>> In 6.3/i386 I have the same problem, but more frequently.
>>>>>>> Date: Sat, 17 Aug 2019 23:03:56 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 01:37:50 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 04:12:31 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 06:46:25 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 09:20:22 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 11:59:08 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 14:34:38 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 17:12:57 +0200 (CEST)
>>>>>>> Date: Sun, 18 Aug 2019 19:47:16 +0200 (CEST)
>>>>>>>
>>>>>>> Do I have any bugs/deficiencies in my configs, missed something?
>>>>>>> Is there any way to make it work uninterruptedly?
>>>>>>> I would be very greatful if you could help me with this case.
>>>>>>>
>>>>>>> $cat /etc/hostname.enc0
>>>>>>> up
>>>>>>>
>>>>>>> $cat /etc/hostname.vr3
>>>>>>> inet 10.0.17.254 255.255.255.0 NONE description "LAN17"
>>>>>>> group trust
>>>>>>>
>>>>>>> $cat /etc/iked.conf
>>>>>>> local_gw_RAC17 = "10.0.17.254" # lan_RAC
>>>>>>> local_lan_RAC17 = "10.0.17.0/24"
>>>>>>> remote_gw_MON = "1.2.3.5" # fw_MON
>>>>>>> remote_lan_MON = "172.16.1.0/24"
>>>>>>> ikev2 quick active esp \
>>>>>>> from $local_gw_RAC17 to $remote_gw_MON \
>>>>>>> from $local_lan_RAC17 to $remote_lan_MON peer $remote_gw_MON \
>>>>>>> childsa enc chacha20-poly1305 \
>>>>>>> psk "psk"
>>>>>>>
>>>>>>> $cat /etc/pf.conf
>>>>>>> # RAC-fwTEST
>>>>>>> ext_if = "vr0"
>>>>>>> lan_rac_if = "vr3" # vr3 -
>>>>>>> lan_rac_local = $lan_rac_if:network # 10.0.17.0/24
>>>>>>> backup_if = "vr2" # vr2 - lewy port
>>>>>>> backup_local = $backup_if:network # 10.0.117/24
>>>>>>>
>>>>>>> bud = "1.2.3.0/25"
>>>>>>> rdk_wy = "1.2.3.4"
>>>>>>> rdk_mon = "1.2.3.5"
>>>>>>> panac_krz = "1.2.3.6"
>>>>>>> panac_rac = "1.2.3.7"
>>>>>>>
>>>>>>> set fingerprints "/dev/null"
>>>>>>> set skip on { lo, enc0 }
>>>>>>> set block-policy drop
>>>>>>> set optimization normal
>>>>>>> set ruleset-optimization basic
>>>>>>>
>>>>>>> antispoof quick for {lo0, $lan_rac_if, $backup_if }
>>>>>>>
>>>>>>> match out log on $ext_if from { $lan_rac_local, $backup_local } nat-to
>>>>>>> $ext_if set prio (3, 7)
>>>>>>>
>>>>>>> block all
>>>>>>>
>>>>>>> match in all scrub (no-df random-id)
>>>>>>> match out all scrub (no-df random-id)
>>>>>>> pass out on egress keep state
>>>>>>>
>>>>>>> pass from { 10.0.201.0/24, $lan_rac_local, $backup_local } to any set
>>>>>>> prio (3, 7) keep state
>>>>>>>
>>>>>>> ssh_port = "1071"
>>>>>>> table <ssh_trust> const { $bud, $rdk_wy, $rdk_mon, $panac_krz,
>>>>>>> $panac_rac, 10.0.2.0/24, 10.0.15.0/24, 10.0.100.0/24 }
>>>>>>> table <bruteforce> persist counters
>>>>>>> block from <bruteforce>
>>>>>>> pass in log quick inet proto tcp from <ssh_trust> to $ext_if port
>>>>>>> $ssh_port flags S/SA \
>>>>>>> set prio (7, 7) keep state \
>>>>>>> (max-src-conn 15, max-src-conn-rate 2/10, overload <bruteforce>
>>>>>>> flush global)
>>>>>>>
>>>>>>> icmp_types = "{ echoreq, unreach }"
>>>>>>> pass inet proto icmp all icmp-type $icmp_types \
>>>>>>> set prio (7, 7) keep state
>>>>>>>
>>>>>>> table <vpn_peers> const { $rdk_mon, $panac_rac, $panac_krz }
>>>>>>> pass out quick on egress proto esp from (egress:0) to <vpn_peers>
>>>>>>> set prio (6, 7) keep state
>>>>>>> pass out quick on egress proto udp from (egress:0) to <vpn_peers> port
>>>>>>> {500, 4500} set prio (6, 7) keep state
>>>>>>> pass in quick on egress proto esp from <vpn_peers> to (egress:0)
>>>>>>> set prio (6, 7) keep state
>>>>>>> pass in quick on egress proto udp from <vpn_peers> to (egress:0) port
>>>>>>> {500, 4500} set prio (6, 7) keep state
>>>>>>> pass out quick on trust received-on enc0 set prio (6, 7) keep state
>>>>>>>
>>>>>>> pass in on egress proto udp from any to (egress:0) port
>>>>>>> {isakmp,ipsec-nat-t} set prio (6,7) keep state
>>>>>>> pass in on egress proto {ah,esp} set prio (6,7) keep state
>>>>>>>
>>>>>>> # By default, do not permit remote connections to X11
>>>>>>> block return in on ! lo0 proto tcp to port 6000:6010
>>>>>>>
>>>>>>> $cat iked_monitor.sh
>>>>>>> #!/bin/sh
>>>>>>> while true
>>>>>>> do
>>>>>>> vpn=`ping -c 3 -w 1 -I 10.0.17.254 172.16.1.254 | grep packets | awk -F
>>>>>>> " " '{print $4}'`
>>>>>>>
>>>>>>> if [ "${vpn}" -eq 0 ] ; then
>>>>>>> mon=`ping -c 3 -w 1 the_other_side_WAN_IP | grep packets | awk -F " "
>>>>>>> '{print $4}'`
>>>>>>> wan=`ping -c 3 -w 1 8.8.8.8 | grep packets | awk -F " " '{print $4}'`
>>>>>>>
>>>>>>> if [ "${mon}" -gt 0 ] && [ "${wan}" -gt 0 ] ; then
>>>>>>> echo vpn: ${vpn}, mon: ${mon}, wan: ${wan} | mail -s "no ping
>>>>>>> through VPN RACTEST-MON! restartng iked!" em...@example.com
>>>>>>> rcctl restart iked
>>>>>>> fi
>>>>>>> fi
>>>>>>> sleep 32
>>>>>>> done
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Radek
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Radek
>>>>>
>>>>
>>>
>>>
>>> --
>>> Radek
>>>
>>
>
>
> --
> Radek
