Were you able to resolve this issue?
> On Jun 10, 2020, at 15:59, obs...@loopw.com wrote:
>
> I have a small fleet of protectli firewalls, all of them with em nics. Only
> the units i’ve upgraded to 6.7 are showing interface errors, where 6.6 is
> definitely not.
>
>
>> On Jun 8, 2020, at 5:30 PM, Gabri Tofano <ga...@tofanos.com> wrote:
>>
>> Hi all,
>>
>> I'm sending this e-mail since I have found other users in this mailing-list
>> using the same device without issues.
>>
>> I'm using a "Protectli FW1" with FreeBSD 12.1 amd64 as a firewall which is
>> serving me with great performances and no issues at all. The appliance has 4
>> Intel Gigabit 82583V Ethernet NIC ports which are working very well under
>> FreeBSD 12.1. I have used PFsense as well prior to FreeBSD and it worked
>> without issues too.
>>
>> I took the decision to move to OpenBSD 6.7 amd64 in order to benefit of the
>> latest pf (and other) features but unfortunately the OS is giving me an
>> issue which I guess is related to the NIC drivers; When I was connected via
>> ssh I felt some glitches meanwhile I was typing/moving around with the
>> editor, so I started to ping the inside interface from my wired connected pc
>> and found out that time to time the appliance is responding with a 100+/200+
>> ms response (I have cut some 1ms reply to make it shorter):
>>
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=163ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=2ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=3ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=43ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time<1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=4ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=1ms TTL=254
>> Reply from 172.16.200.1: bytes=32 time=257ms TTL=254
>>
>> With FreeBSD 12.1 is steady at <1/1ms all the time and even under load.
>>
>> As an online gamer as well, I felt the glitches meanwhile playing few online
>> FPS games using OpenBSD 6.7 on the appliance. Looking at the interface
>> statistics on OpenBSD I found out that inbound/outbound errors are present
>> (this has been taken after few minutes of a reinstall to test it again):
>>
>> FRW-FW1# netstat -i
>> Name Mtu Network Address Ipkts Ifail Opkts Ofail
>> Colls
>> em0 1500 <Link> xx:xx:xx:xx:xx:xx 1317600 2351 466114 0
>> 0
>> em0 1500 74.215.235/ xxx.xxx.xxx.xxx 1317600 2351 466114 0
>> 0
>> em1 1500 <Link> xx:xx:xx:xx:xx:xx 392782 18 1199871 1
>> 0
>> em1 1500 172.16.200. 172.16.200.1 392782 18 1199871 1
>> 0
>> em2 1500 <Link> xx:xx:xx:xx:xx:xx 156 0 55 1
>> 0
>> em2 1500 172.16.103/ 172.16.103.254 156 0 55 1
>> 0
>> em3* 1500 <Link> xx:xx:xx:xx:xx:xx 0 0 0 0
>> 0
>> enc0* 0 <Link> 0 0 0 0
>> 0
>> pflog0 33136 <Link> 0 0 0 0
>> 0
>>
>> Looking at the Cisco 3560G where the ports are connected there are no errors
>> at all. I have also doublechecked the drivers and the firmware installed by
>> fw_update are the following:
>>
>> vmm-firmware-1.11.0p2
>> inteldrm-firmware-20181218
>> intel-firmware-20200508v0
>>
>> I have done multiple reinstall with different OS to make sure that this is
>> related to OpenBSD 6.7 itself and found the following:
>>
>> PFsense 2.4.5: no issues at all
>> FreeBSD 12.1: no issues at all
>> OPNsense: interface errors
>> OpenBSD: interface errors and interface latency spikes
>>
>> I have also swapped the ethernet cables and contacted Protectli which has
>> confirmed that this appliance has been tested on OpenBSD (it looks like 6.3).
>>
>> Here the dmesg output:
>>
>> OpenBSD 6.7 (GENERIC.MP) #2: Thu Jun 4 09:55:08 MDT 2020
>>
>> r...@syspatch-67-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>> real mem = 4163854336 (3970MB)
>> avail mem = 4025044992 (3838MB)
>> mpath0 at root
>> scsibus0 at mpath0: 256 targets
>> mainbus0 at root
>> bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xecea0 (51 entries)
>> bios0: vendor American Megatrends Inc. version "5.6.5" date 10/24/2018
>> bios0: Protectli FW1
>> acpi0 at bios0: ACPI 5.0
>> acpi0: sleep states S0 S3 S4 S5
>> acpi0: tables DSDT FACP APIC FPDT FIDT MCFG LPIT HPET SSDT SSDT SSDT UEFI
>> acpi0: wakeup devices PS2K(S3) PS2M(S3) XHC1(S4) RP01(S4) PXSX(S4) RP02(S4)
>> PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) BRCM(S0)
>> acpitimer0 at acpi0: 3579545 Hz, 24 bits
>> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>> cpu0 at mainbus0: apid 0 (boot processor)
>> cpu0: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, 2000.47 MHz, 06-37-09
>> cpu0:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
>> cpu0: 1MB 64b/line 16-way L2 cache
>> cpu0: smt 0, core 0, package 0
>> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
>> cpu0: apic clock running at 83MHz
>> cpu0: mwait min=64, max=64, C-substates=0.2.0.0.0.0.3.3, IBE
>> cpu1 at mainbus0: apid 2 (application processor)
>> cpu1: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, 2000.01 MHz, 06-37-09
>> cpu1:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
>> cpu1: 1MB 64b/line 16-way L2 cache
>> cpu1: smt 0, core 1, package 0
>> cpu2 at mainbus0: apid 4 (application processor)
>> cpu2: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, 2000.03 MHz, 06-37-09
>> cpu2:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
>> cpu2: 1MB 64b/line 16-way L2 cache
>> cpu2: smt 0, core 2, package 0
>> cpu3 at mainbus0: apid 6 (application processor)
>> cpu3: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz, 2000.01 MHz, 06-37-09
>> cpu3:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,RDRAND,NXE,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,TSC_ADJUST,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
>> cpu3: 1MB 64b/line 16-way L2 cache
>> cpu3: smt 0, core 3, package 0
>> ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 87 pins
>> acpimcfg0 at acpi0
>> acpimcfg0: addr 0xe0000000, bus 0-255
>> acpihpet0 at acpi0: 14318179 Hz
>> acpiprt0 at acpi0: bus 0 (PCI0)
>> acpiprt1 at acpi0: bus 1 (RP01)
>> acpiprt2 at acpi0: bus 2 (RP02)
>> acpiprt3 at acpi0: bus 3 (RP03)
>> acpiprt4 at acpi0: bus 4 (RP04)
>> acpiec0 at acpi0: not present
>> acpicpu0 at acpi0: C3(10@1500 mwait.1@0x52), C2(10@500 mwait.1@0x51),
>> C1(1000@1 mwait.1), PSS
>> acpicpu1 at acpi0: C3(10@1500 mwait.1@0x52), C2(10@500 mwait.1@0x51),
>> C1(1000@1 mwait.1), PSS
>> acpicpu2 at acpi0: C3(10@1500 mwait.1@0x52), C2(10@500 mwait.1@0x51),
>> C1(1000@1 mwait.1), PSS
>> acpicpu3 at acpi0: C3(10@1500 mwait.1@0x52), C2(10@500 mwait.1@0x51),
>> C1(1000@1 mwait.1), PSS
>> acpipwrres0 at acpi0: PLPE
>> acpipwrres1 at acpi0: PLPE
>> acpipwrres2 at acpi0: USBC, resource for EHC1, OTG1
>> acpipwrres3 at acpi0: CLK0, resource for CAM1
>> acpipwrres4 at acpi0: CLK1, resource for CAM0, CAM2
>> acpicmos0 at acpi0
>> acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
>> "DMA0F28" at acpi0 not configured
>> acpibtn0 at acpi0: SLPB
>> "BCM2E1A" at acpi0 not configured
>> "BCM4752" at acpi0 not configured
>> "INTCF0B" at acpi0 not configured
>> "INTCF1A" at acpi0 not configured
>> "INTCF1C" at acpi0 not configured
>> "SMO91D0" at acpi0 not configured
>> "ATML1000" at acpi0 not configured
>> "ATML2000" at acpi0 not configured
>> "INT33BD" at acpi0 not configured
>> acpivideo0 at acpi0: GFX0
>> acpivout0 at acpivideo0: DD1F
>> cpu0: using VERW MDS workaround
>> cpu0: Enhanced SpeedStep 2000 MHz: speeds: 1993, 1992, 1909, 1826, 1743,
>> 1660, 1577, 1494, 1411, 1328 MHz
>> pci0 at mainbus0 bus 0
>> pchb0 at pci0 dev 0 function 0 "Intel Bay Trail Host" rev 0x11
>> inteldrm0 at pci0 dev 2 function 0 "Intel Bay Trail Video" rev 0x11
>> drm0 at inteldrm0
>> inteldrm0: msi, VALLEYVIEW, gen 7
>> ahci0 at pci0 dev 19 function 0 "Intel Bay Trail AHCI" rev 0x11: msi, AHCI
>> 1.3
>> ahci0: port 0: 3.0Gb/s
>> scsibus1 at ahci0: 32 targets
>> sd0 at scsibus1 targ 0 lun 0: <ATA, Hoodisk SSD, SBFM> naa.0000000000000000
>> sd0: 15272MB, 512 bytes/sector, 31277232 sectors, thin
>> xhci0 at pci0 dev 20 function 0 "Intel Bay Trail xHCI" rev 0x11: msi, xHCI
>> 1.0
>> usb0 at xhci0: USB revision 3.0
>> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
>> 3.00/1.00 addr 1
>> "Intel Bay Trail TXE" rev 0x11 at pci0 dev 26 function 0 not configured
>> azalia0 at pci0 dev 27 function 0 "Intel Bay Trail HD Audio" rev 0x11: msi
>> azalia0: no supported codecs
>> ppb0 at pci0 dev 28 function 0 "Intel Bay Trail PCIE" rev 0x11: msi
>> pci1 at ppb0 bus 1
>> em0 at pci1 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
>> xx:xx:xx:xx:xx:xx
>> ppb1 at pci0 dev 28 function 1 "Intel Bay Trail PCIE" rev 0x11: msi
>> pci2 at ppb1 bus 2
>> em1 at pci2 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
>> xx:xx:xx:xx:xx:xx
>> ppb2 at pci0 dev 28 function 2 "Intel Bay Trail PCIE" rev 0x11: msi
>> pci3 at ppb2 bus 3
>> em2 at pci3 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
>> xx:xx:xx:xx:xx:xx
>> ppb3 at pci0 dev 28 function 3 "Intel Bay Trail PCIE" rev 0x11: msi
>> pci4 at ppb3 bus 4
>> em3 at pci4 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
>> xx:xx:xx:xx:xx:xx
>> pcib0 at pci0 dev 31 function 0 "Intel Bay Trail LPC" rev 0x11
>> ichiic0 at pci0 dev 31 function 3 "Intel Bay Trail SMBus" rev 0x11: apic 1
>> int 18
>> iic0 at ichiic0
>> spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-12800 SO-DIMM
>> isa0 at pcib0
>> isadma0 at isa0
>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>> pckbd0 at pckbc0 (kbd slot)
>> wskbd0 at pckbd0: console keyboard
>> pcppi0 at isa0 port 0x61
>> spkr0 at pcppi0
>> it0 at isa0 port 0x2e/2: IT8772F rev 1, EC port 0xa40
>> vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
>> vscsi0 at root
>> scsibus2 at vscsi0: 256 targets
>> softraid0 at root
>> scsibus3 at softraid0: 256 targets
>> root on sd0a (78fa67e12e212447.a) swap on sd0b dump on sd0b
>> inteldrm0: 1024x768, 32bpp
>> wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using wskbd0
>> wsdisplay0: screen 1-5 added (std, vt100 emulation)
>>
>> Any clue of what the issue could be? I have reinstalled OSs for 1 week now
>> to make all the tests and, even if I would have not any issues in going back
>> with FreeBSD 12.1, I just would hate to not have the chance to run OpenBSD
>> 6.7 with all the newer features.
>>
>> I'm using OpenBSD 6.7 at the moment and see if I can get this issue resolved
>> soon, otherwise I need to reinstall FreeBSD 12.1 again to have a steady
>> network connection.
>>
>> Thanks!
>> Gabri
>
