Hello,
I've been trying to track down what exactly is causing such a large increase
in traffic on pfsync(4) interfaces on several of the firewall pairs I've
upgraded to 7.4. All have recent errata applied.
custedge1$ doas syspatch -l
002_msplit
003_patch
004_ospfd
005_tmux
006_httpd
007_perl
008_vmm
009_pf
011_ssh
I've seen both CPU graphs and network graphs jump greatly since the upgrades,
by about 3-4x in some cases.
I've included some data in-line, but due to production nature of these
for pcap data and pfctl -ss -vv outputs I sent directly to dlg.
The issue wasn't noticed immediately, and even now for the most part everything
is performing 'OK' in that resources behind these firewalls are loading fine
and latency is low.
This has happened on a few virtualized fw pairs running on proxmox nodes
with virtio-backed vio(4) interfaces that tend to be like so:
- one vio for 'outside'
- one vio for 'inside'
- one vio for pfsync/also ssh pf conf changes to secondary
All generally have lots of vlan/carp interfaces.
However, I also have a physical pair of fw that use em(4) and the pfsync there
is also showing these same symptoms, linked directly to each other with a single
cable.
Here is a bwm-ng output of an affected host, the pfsync interface
is yarding around data at a higher rate vs every other interface (this
time of day is a quieter time)
bwm-ng v0.6.3 (probing every 0.500s), press 'h' for help
input: getifaddrs type: rate
| iface Rx Tx Total
==============================================================================
lo0: 0.00 b/s 0.00 b/s 0.00 b/s
vio0: 630.04 kb/s 1.13 Mb/s 1.76 Mb/s
vio1: 27.78 Mb/s 28.12 Mb/s 55.90 Mb/s
vio2: 1.47 Mb/s 199.67 kb/s 1.67 Mb/s
carp2100: 0.00 b/s 1.10 kb/s 1.10 kb/s
carp600: 0.00 b/s 1.10 kb/s 1.10 kb/s
carp601: 1.10 kb/s 1.10 kb/s 2.19 kb/s
carp602: 0.00 b/s 1.10 kb/s 1.10 kb/s
carp605: 0.00 b/s 1.10 kb/s 1.10 kb/s
carp607: 16.05 kb/s 1.10 kb/s 17.14 kb/s
carp612: 0.00 b/s 1.10 kb/s 1.10 kb/s
carp614: 0.00 b/s 1.10 kb/s 1.10 kb/s
carp615: 0.00 b/s 1.10 kb/s 1.10 kb/s
carp99: 1.06 Mb/s 1.10 kb/s 1.06 Mb/s
pfsync0: 25.35 Mb/s 27.17 Mb/s 52.52 Mb/s
vlan2100: 0.00 b/s 1.10 kb/s 1.10 kb/s
vlan600: 0.00 b/s 1.10 kb/s 1.10 kb/s
vlan601: 1.10 kb/s 1.10 kb/s 2.19 kb/s
vlan602: 0.00 b/s 1.10 kb/s 1.10 kb/s
vlan605: 0.00 b/s 1.10 kb/s 1.10 kb/s
vlan607: 16.05 kb/s 82.25 kb/s 98.30 kb/s
vlan612: 0.00 b/s 1.10 kb/s 1.10 kb/s
vlan614: 0.00 b/s 1.10 kb/s 1.10 kb/s
vlan615: 0.00 b/s 1.10 kb/s 1.10 kb/s
vlan99: 1.06 Mb/s 102.76 kb/s 1.16 Mb/s
lo1: 0.00 b/s 0.00 b/s 0.00 b/s
pflog0: 0.00 b/s 3.32 kb/s 3.32 kb/s
vlan616: 222.81 kb/s 1.12 Mb/s 1.35 Mb/s
carp616: 192.00 kb/s 1.10 kb/s 193.10 kb/s
------------------------------------------------------------------------------
total: 57.80 Mb/s 57.95 Mb/s 115.75 Mb/s
custedge1$ ifconfig vio1
vio1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 82:7b:8a:de:50:6e
index 2 priority 0 llprio 3
media: Ethernet autoselect
status: active
inet 10.100.1.5 netmask 0xfffffffc broadcast 10.100.1.7
custedge1$ ifconfig pfsync0
pfsync0: flags=41<UP,RUNNING> mtu 1500
index 18 priority 0 llprio 3
encap: parent vio1
pfsync: syncdev: vio1 syncpeer: 10.100.1.6 maxupd: 128 defer: off
groups: carp pfsync
I've verified maxupd is the same on both sides.
I recently switched to using syncpeer to see if that would lessen the load,
it does appear to have helped but I am suspecting over time the amount of
duplicates (maybe?) increases linearly. It seemed like downing the pfsync if
and then upping it 'reset' the issue for awhile, at least with much lower than
what I saw the other day (>100mbit.. on pfsync). I have some links to some
librenms graphs below as well that show this in a generalized graph...
https://ryan.slipgate.org/bandwidth-pfsync-increase.png
https://ryan.slipgate.org/bandwidth-reduce-after-syncpeer.png
- this one is actually from proxmox but shows the initial reduction after
changing
to syncpeer, but might just be from 'down/up'
https://ryan.slipgate.org/cpu-increase.png
- for cpu increase, might be more of a red herring as I've noticed cpu increases
across releases every time unlocking happens, and top doesn't seem to support
the graph showing such an increase.... a small drop happened after syncpeer
changes though. just not back to pre upgrade levels. accounting oddities?
I found one other case of this reported here:
https://marc.info/?l=openbsd-misc&m=170172354625962&w=2
Thanks a bunch to IcePic, double-p and sthen for suggestions on IRC to help
gather the best info. vmstat vm output and dmesg follows
If I can provide more data, I will. Thanks!
-Ryan
1 users Load 0.81 0.81 0.79 custedge1. 01:19:47
memory totals (in KB) PAGING SWAPPING Interrupts
real virtual free in out in out 13280 total
Active 48208 48208 819820 ops uhci0
All 1186816 1186816 3152828 pages 1 virtio1
410 virtio2
Proc:r d s w Csw Trp Sys Int Sof Flt forks 4293 virtio3
1 66 17039 15 351 5058 104 17 fkppw 352 virtio4
fksvm fdc0
4.1%Int 1.3%Spn 21.7%Sys 0.0%Usr 72.9%Idle pwait 400 clock
| | | | | | | | | | | relck 7824 ipi
||@=========== rlkok
noram
Namei Sys-cache Proc-cache No-cache 1 ndcpy
Calls hits % hits % miss % fltcp
11 11 100 3 zfod
cow
Disks cd0 sd0 fd0 16721 fmin
seeks 22294 ftarg
xfers 1 itarg
speed 11K 2 wired
sec 0.0 pdfre
pdscn
pzidl 19902 IPKTS
12 kmape 17941 OPKTS
OpenBSD 7.4 (GENERIC.MP) #2: Fri Dec 8 15:39:04 MST 2023
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2130550784 (2031MB)
avail mem = 2046312448 (1951MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf58d0 (9 entries)
bios0: vendor SeaBIOS version "rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org"
date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC SSDT HPET WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Common KVM processor, 2261.87 MHz, 0f-06-01
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,AES,HV,NXE,LONG,LAHF,MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Common KVM processor, 2262.01 MHz, 0f-06-01
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,AES,HV,NXE,LONG,LAHF,MELTDOWN
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
"QEMUVGID" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0
wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio0
virtio0: apic 0 int 11
virtio1 at pci0 dev 10 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 1 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 32768MB, 512 bytes/sector, 67108864 sectors
virtio1: msix per-VQ
virtio2 at pci0 dev 18 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio2: address 82:53:a5:f0:33:1b
virtio2: msix shared
virtio3 at pci0 dev 19 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio3: address 82:7b:8a:de:50:6e
virtio3: msix shared
virtio4 at pci0 dev 20 function 0 "Qumranet Virtio Network" rev 0x00
vio2 at virtio4: address 22:4f:40:94:5f:28
virtio4: msix shared
ppb0 at pci0 dev 30 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci1 at ppb0 bus 1
ppb1 at pci0 dev 31 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci2 at ppb1 bus 2
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00
addr 1
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev
2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (5032b822eaa07830.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown
carp2100: state transition: BACKUP -> MASTER
carp600: state transition: BACKUP -> MASTER
carp601: state transition: BACKUP -> MASTER
carp602: state transition: BACKUP -> MASTER
carp605: state transition: BACKUP -> MASTER
carp607: state transition: BACKUP -> MASTER
carp611: state transition: BACKUP -> MASTER
carp612: state transition: BACKUP -> MASTER
carp614: state transition: BACKUP -> MASTER
carp615: state transition: BACKUP -> MASTER
carp710: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
carp2100: state transition: MASTER -> BACKUP
carp600: state transition: MASTER -> BACKUP
carp601: state transition: MASTER -> BACKUP
carp602: state transition: MASTER -> BACKUP
carp605: state transition: MASTER -> BACKUP
carp607: state transition: MASTER -> BACKUP
carp611: state transition: MASTER -> BACKUP
carp612: state transition: MASTER -> BACKUP
carp614: state transition: MASTER -> BACKUP
carp615: state transition: MASTER -> BACKUP
carp710: state transition: MASTER -> BACKUP
carp99: state transition: MASTER -> BACKUP
carp2100: state transition: BACKUP -> MASTER
carp600: state transition: BACKUP -> MASTER
carp601: state transition: BACKUP -> MASTER
carp602: state transition: BACKUP -> MASTER
carp605: state transition: BACKUP -> MASTER
carp607: state transition: BACKUP -> MASTER
carp611: state transition: BACKUP -> MASTER
carp612: state transition: BACKUP -> MASTER
carp614: state transition: BACKUP -> MASTER
carp615: state transition: BACKUP -> MASTER
carp710: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
carp2100: state transition: MASTER -> BACKUP
carp600: state transition: MASTER -> BACKUP
carp601: state transition: MASTER -> BACKUP
carp602: state transition: MASTER -> BACKUP
carp605: state transition: MASTER -> BACKUP
carp607: state transition: MASTER -> BACKUP
carp611: state transition: MASTER -> BACKUP
carp612: state transition: MASTER -> BACKUP
carp614: state transition: MASTER -> BACKUP
carp615: state transition: MASTER -> BACKUP
carp710: state transition: MASTER -> BACKUP
carp99: state transition: MASTER -> BACKUP
carp2100: state transition: BACKUP -> MASTER
carp600: state transition: BACKUP -> MASTER
carp601: state transition: BACKUP -> MASTER
carp602: state transition: BACKUP -> MASTER
carp605: state transition: BACKUP -> MASTER
carp607: state transition: BACKUP -> MASTER
carp611: state transition: BACKUP -> MASTER
carp612: state transition: BACKUP -> MASTER
carp614: state transition: BACKUP -> MASTER
carp615: state transition: BACKUP -> MASTER
carp710: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
carp616: state transition: BACKUP -> MASTER
carp616: state transition: MASTER -> BACKUP
carp2100: state transition: MASTER -> BACKUP
carp600: state transition: MASTER -> BACKUP
carp601: state transition: MASTER -> BACKUP
carp602: state transition: MASTER -> BACKUP
carp605: state transition: MASTER -> BACKUP
carp607: state transition: MASTER -> BACKUP
carp612: state transition: MASTER -> BACKUP
carp614: state transition: MASTER -> BACKUP
carp615: state transition: MASTER -> BACKUP
carp99: state transition: MASTER -> BACKUP
carp2100: state transition: BACKUP -> MASTER
carp600: state transition: BACKUP -> MASTER
carp616: state transition: BACKUP -> MASTER
carp601: state transition: BACKUP -> MASTER
carp602: state transition: BACKUP -> MASTER
carp605: state transition: BACKUP -> MASTER
carp607: state transition: BACKUP -> MASTER
carp612: state transition: BACKUP -> MASTER
carp614: state transition: BACKUP -> MASTER
carp615: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
carp2100: state transition: MASTER -> BACKUP
carp600: state transition: MASTER -> BACKUP
carp601: state transition: MASTER -> BACKUP
carp602: state transition: MASTER -> BACKUP
carp605: state transition: MASTER -> BACKUP
carp616: state transition: MASTER -> BACKUP
carp607: state transition: MASTER -> BACKUP
carp612: state transition: MASTER -> BACKUP
carp614: state transition: MASTER -> BACKUP
carp615: state transition: MASTER -> BACKUP
carp99: state transition: MASTER -> BACKUP
carp616: state transition: BACKUP -> MASTER
carp2100: state transition: BACKUP -> MASTER
carp600: state transition: BACKUP -> MASTER
carp601: state transition: BACKUP -> MASTER
carp602: state transition: BACKUP -> MASTER
carp605: state transition: BACKUP -> MASTER
carp607: state transition: BACKUP -> MASTER
carp612: state transition: BACKUP -> MASTER
carp614: state transition: BACKUP -> MASTER
carp615: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
carp2100: state transition: MASTER -> BACKUP
carp600: state transition: MASTER -> BACKUP
carp601: state transition: MASTER -> BACKUP
carp602: state transition: MASTER -> BACKUP
carp605: state transition: MASTER -> BACKUP
carp607: state transition: MASTER -> BACKUP
carp612: state transition: MASTER -> BACKUP
carp614: state transition: MASTER -> BACKUP
carp615: state transition: MASTER -> BACKUP
carp616: state transition: MASTER -> BACKUP
carp99: state transition: MASTER -> BACKUP
carp2100: state transition: BACKUP -> MASTER
carp616: state transition: BACKUP -> MASTER
carp600: state transition: BACKUP -> MASTER
carp601: state transition: BACKUP -> MASTER
carp602: state transition: BACKUP -> MASTER
carp605: state transition: BACKUP -> MASTER
carp607: state transition: BACKUP -> MASTER
carp612: state transition: BACKUP -> MASTER
carp614: state transition: BACKUP -> MASTER
carp615: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
