Hello, I have issues with firewall lags while there is peak in match rule counter in pf. Normally it has match ratio of about 1500/sec, but several times a day it jumps to somewhere around 6k/sec and firewall lags, some traffic gets dropped. This takes a few seconds.
Lag causes system to delay sending carp packets and sometimes backup box promotes itself to master and immediately back to backup. Sadly, after sending inverse ARP. I workarounded this issue by setting advbase to 10. Another problem is obviously with normal forwarding traffic, like lags in online games or iptv streams. There is no visible raise in cpu utilization, but cpu load goes from about 0.7 to 1.5 and there are packets getting dropped on wan interface. Box is Core i3 530 on Supermicro X8SIL with 2x1GB RAM, intel 40GB SSD, two 82574 and two 82571 NICs. In afternoon hours it is loaded on 40k/25k tx/rx pps on wan interface. Looking to systat vmstat, LAN and WAN nics are getting around 7.5k interrupts, while pfsync about 2.5-3k and interrupts in top take about 60-70%. I tried to switch NICs for i350, but it had no effect, same thing with openBSD versions, 5.6 5.7 and 5.8 have same behavior. I also tried to replacing other hardware like CPU for Xeon X3430 or motherboard S5500BC with Xeon E5620, but without effect. Happens also on backup box when it runs as master (same hw config). System is running GENERIC.MP stable amd64 kernel. I read in some discussions, that raising interrupt limit and rx/tx queue in em(4) driver or using broadcoms instead of intels might help, but didnt try it yet. Is there any way to determine what is causing the peaks and how to prevent them or getting system powerful enough to handle them? pfctl -si Status: Enabled for 0 days 22:12:20 Debug: err State Table Total Rate current entries 66901 searches 5003330275 62588.6/s inserts 47704143 596.7/s removals 47637242 595.9/s Counters match 96819915 1211.2/s bad-offset 0 0.0/s fragment 1850 0.0/s short 86 0.0/s normalize 48 0.0/s memory 786228 9.8/s bad-timestamp 0 0.0/s congestion 3948624 49.4/s ip-option 24341 0.3/s proto-cksum 0 0.0/s state-mismatch 1644853 20.6/s state-insert 464 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 3948 0.0/s translate 0 0.0/s no-route 0 0.0/s kern.netlivelocks=1534 netstat -si em0 1500 <Link> 1533962428 266567 955232172 0 0 em1 1500 <Link> 979515291 8697 1526507571 0 0 em2 1500 <Link> 6970941 0 140093911 0 0 em3* 1500 <Link> 0 0 0 0 0 OpenBSD 5.8-stable (GENERIC.MP) #1: Sun Nov 15 17:29:19 CET 2015 :/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2121859072 (2023MB) avail mem = 2053718016 (1958MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.6 @ 0x9f000 (68 entries) bios0: vendor American Megatrends Inc. version "1.1" date 05/27/2010 bios0: Supermicro X8SIL acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC MCFG OEMB HPET GSCI DMAR SSDT EINJ BERT ERST HEST acpi0: wakeup devices P0P1(S4) P0P3(S4) P0P4(S4) P0P5(S4) P0P6(S4) BR1E(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4) USB3(S4) USB4(S4) USB5(S4) U SB6(S4) GBE_(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz, 2933.75 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL ,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 133MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 4 (application processor) cpu1: Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz, 2933.34 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL ,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 2, package 0 cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz, 2933.34 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL ,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 1, core 0, package 0 cpu3 at mainbus0: apid 5 (application processor) cpu3: Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz, 2933.34 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL ,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 1, core 2, package 0 ioapic0 at mainbus0: apid 6 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 1, remapped to apid 6 acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (P0P1) acpiprt2 at acpi0: bus -1 (P0P3) acpiprt3 at acpi0: bus -1 (P0P5) acpiprt4 at acpi0: bus -1 (P0P6) acpiprt5 at acpi0: bus 4 (BR1E) acpiprt6 at acpi0: bus 1 (BR20) acpiprt7 at acpi0: bus 2 (BR24) acpiprt8 at acpi0: bus 3 (BR25) acpicpu0 at acpi0: C3(350@17 mwait.1@0x20), C2(500@17 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpicpu1 at acpi0: C3(350@17 mwait.1@0x20), C2(500@17 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpicpu2 at acpi0: C3(350@17 mwait.1@0x20), C2(500@17 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpicpu3 at acpi0: C3(350@17 mwait.1@0x20), C2(500@17 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpibtn0 at acpi0: SLPB acpibtn1 at acpi0: PWRB ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 2933 MHz: speeds: 2933, 2800, 2667, 2533, 2400, 2267, 2133, 2000, 1867, 1733, 1600, 1467, 1333, 1200 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel Core Host" rev 0x12 ehci0 at pci0 dev 26 function 0 "Intel 3400 USB" rev 0x05: apic 6 int 21 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb0 at pci0 dev 28 function 0 "Intel 3400 PCIE" rev 0x05: msi pci1 at ppb0 bus 1 em0 at pci1 dev 0 function 0 "Intel 82571EB" rev 0x06: apic 6 int 16, address 00:1b:78:57:c4:ea em1 at pci1 dev 0 function 1 "Intel 82571EB" rev 0x06: apic 6 int 17, address 00:1b:78:57:c4:eb ppb1 at pci0 dev 28 function 4 "Intel 3400 PCIE" rev 0x05: msi pci2 at ppb1 bus 2 em2 at pci2 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address 00:25:90:0d:24:e2 ppb2 at pci0 dev 28 function 5 "Intel 3400 PCIE" rev 0x05: msi pci3 at ppb2 bus 3 em3 at pci3 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address 00:25:90:0d:24:e3 ehci1 at pci0 dev 29 function 0 "Intel 3400 USB" rev 0x05: apic 6 int 23 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5 pci4 at ppb3 bus 4 vga1 at pci4 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 31 function 0 "Intel 3420 LPC" rev 0x05 pciide0 at pci0 dev 31 function 2 "Intel 3400 SATA" rev 0x05: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI pciide0: using apic 6 int 19 for native-PCI interrupt wd0 at pciide0 channel 0 drive 0: <INTEL SSDSA2BT040G3> wd0: 16-sector PIO, LBA48, 38166MB, 78165360 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6 ichiic0 at pci0 dev 31 function 3 "Intel 3400 SMBus" rev 0x05: apic 6 int 18 iic0 at ichiic0 sdtemp0 at iic0 addr 0x18: stts424e02 sdtemp1 at iic0 addr 0x1a: stts424e02 spdmem0 at iic0 addr 0x50: 1GB DDR3 SDRAM ECC PC3-10600 with thermal sensor spdmem1 at iic0 addr 0x52: 1GB DDR3 SDRAM ECC PC3-10600 with thermal sensor pciide1 at pci0 dev 31 function 5 "Intel 3400 SATA" rev 0x05: DMA, channel 0 wired to native-PCI, channel 1 wired to native-PCI pciide1: using apic 6 int 19 for native-PCI interrupt isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 wbsio0 at isa0 port 0x2e/2: W83627DHG rev 0x25 lm1 at wbsio0 port 0xa10/8: W83627DHG uhub2 at uhub0 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2 uhidev0 at uhub2 port 2 configuration 1 interface 0 "Winbond Electronics Corp Hermon USB hidmouse Device" rev 1.10/0.01 addr 3 uhidev0: iclass 3/1 ums0 at uhidev0: 3 buttons, Z dir wsmouse0 at ums0 mux 0 uhidev1 at uhub2 port 2 configuration 1 interface 1 "Winbond Electronics Corp Hermon USB hidmouse Device" rev 1.10/0.01 addr 3 uhidev1: iclass 3/1 ukbd0 at uhidev1: 8 variable keys, 6 key codes wskbd1 at ukbd0 mux 1 wskbd1: connecting to wsdisplay0 uhub3 at uhub1 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2 vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets root on wd0a (6493b8f65ac66d12.a) swap on wd0b dump on wd0b carp: carp1 demoted group carp by 1 to 129 (carpdev) carp: carp99 demoted group carp by 1 to 130 (carpdev) carp: pfsync0 demoted group carp by 32 to 162 (pfsync init) carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init) carp: pfsync0 demoted group carp by 1 to 163 (pfsync bulk start) carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start) Regards, Martin Hlavaty
