Hello,
I'm running 6.6 Stable amd64 on an 4 Core VM with ix(4) PCI-E SRIOV
attached Nics
Im seeing performance of about 450Mb/s throughput
it bridges /aggregates about 90 vlans into one physical 10G port (and
isolates them (with all the vlans in the bridge and in the same
protection
im seeing alot of of packets dropped because of destination
unreachable ( about 3K destinations unreachable per second)
my guess is that there are bursts of packets coming through and the
cpu cant process them in time... Im wondering is there a handy way of
increasing buffers to smooth out these micro bursts ? to see if this
will improve the situation
with the system running about 20 hours the following stats are
# netstat -nI bridge101 ;netstat -nI ix0;netstat -nI ix1;netstat -nI ix2
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
bridge101 1500 <Link> 3795904105 0
3933822265 44817 0
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
ix0 1700 <Link> 00:12:c0:88:07:b8 2436105363 0
1377058248 44999 0
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
ix1 1708 <Link> 00:12:c0:88:07:b9 297914355 0 377157134
570713 0
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
ix2 1708 <Link> 00:12:c0:88:07:b6 1233726645 0
2173231651 5807022 0
netstat of a random vlans in the bridge
netstat -nI vlan4001
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
vlan4001 1700 <Link> 00:12:c0:88:07:b6 1690797 0 3932075 23385 0
ngabr# netstat -nI vlan4002
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
vlan4002 1700 <Link> 00:12:c0:88:07:b6 5222477 0 8170507 31109 0
ngabr# netstat -nI vlan4003
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
vlan4003 1700 <Link> 00:12:c0:88:07:b6 7333418 0 7842635 23394 0
I found this using the following command to test
netstat -sr;sleep 5;netstat -sr
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
128772097 destinations found unreachable
0 uses of a wildcard route
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
128787229 destinations found unreachable
0 uses of a wildcard route
netstat
output of top command
load averages: 0.81, 0.76, 0.82
ngabr 23:57:04
50 processes: 46 idle, 4 on processor
up 21:45
CPU0 states: 0.0% user, 0.0% nice, 0.8% sys, 0.4% spin, 8.8%
intr, 90.0% idle
CPU1 states: 0.0% user, 0.0% nice, 34.0% sys, 0.8% spin, 0.0%
intr, 65.2% idle
CPU2 states: 0.0% user, 0.0% nice, 40.2% sys, 1.4% spin, 0.0%
intr, 58.4% idle
CPU3 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0%
intr, 100% idle
Memory: Real: 25M/916M act/tot Free: 1050M Cache: 493M Swap: 0K/0K
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
34304 root 10 -20 0K 1184K onproc/1 - 16.8H 74.22% softnet
77304 root 28 0 2000K 3940K onproc/2 - 0:00 0.05% top
70327 root -22 0 0K 1184K onproc/3 - 21.7H 0.00% idle3
82993 root -22 0 0K 1184K onproc/0 - 19.6H 0.00% idle0
93492 root -22 0 0K 1184K sleep/2 - 868:12 0.00% idle2
65795 root -22 0 0K 1184K sleep/1 - 845:29 0.00% idle1
34062 root -22 0 0K 1184K sleep/0 bored 12:00 0.00% softclock
67470 _pflogd 4 0 888K 572K sleep/2 bpf 2:34 0.00% pflogd
16631 root 18 0 0K 1184K sleep/2 syncer 1:07 0.00% update
37166 root 10 0 0K 1184K sleep/1 bored 1:01 0.00% systqmp
64060 fireman 2 0 1404K 2976K sleep/0 select 0:47 0.00% sshd
62514 root 2 0 696K 1284K sleep/1 poll 0:11 0.00% cron
93672 _ntp 2 -20 1244K 2740K idle poll 0:09 0.00% ntpd
0 root -18 0 0K 1184K sleep/0 schedul 0:07 0.00% swapper
74711 _syslogd 2 0 1164K 1512K idle kqread 0:07 0.00% syslogd
2383 root 68 20 0K 1184K sleep/2 pgzero 0:05 0.00% zerothread
85602 root 18 0 1052K 908K sleep/3 pause 0:04 0.00% ksh
58290 root -18 0 0K 1184K idle reaper 0:04 0.00% reaper
32404 _dhcp 2 0 896K 556K idle poll 0:03 0.00% dhclient
62783 root 10 0 0K 1184K idle bored 0:01 0.00% systq
dmesg below
OpenBSD 6.6 (GENERIC.MP) #8: Fri Apr 17 15:06:32 MDT 2020
r...@syspatch-66-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2130526208 (2031MB)
avail mem = 2053304320 (1958MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5920 (9 entries)
bios0: vendor SeaBIOS version
"rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org" date 04/01/2014
bios0: QEMU Standard PC (Q35 + ICH9, 2009)
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC SSDT HPET MCFG
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 581.07 MHz, 06-3e-04
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,FSGSBASE,TSC_ADJUST,SMEP,ERMS,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 616.35 MHz, 06-3e-04
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,FSGSBASE,TSC_ADJUST,SMEP,ERMS,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 618.19 MHz, 06-3e-04
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,FSGSBASE,TSC_ADJUST,SMEP,ERMS,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu2: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu2: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu2: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 628.12 MHz, 06-3e-04
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,FSGSBASE,TSC_ADJUST,SMEP,ERMS,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu3: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu3: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu3: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xb0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
no _STA method
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
"QEMUVGID" at acpi0 not configured
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel 82801I HD Audio" rev 0x03: msi
azalia0: no HD-Audio codecs
ppb0 at pci0 dev 28 function 0 vendor "Red Hat", unknown product
0x000c rev 0x00: apic 0 int 10
pci1 at ppb0 bus 1
ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
00:12:c0:88:07:b8
ppb1 at pci0 dev 28 function 1 vendor "Red Hat", unknown product
0x000c rev 0x00: apic 0 int 10
pci2 at ppb1 bus 2
ix1 at pci2 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
00:12:c0:88:07:b9
ppb2 at pci0 dev 28 function 2 vendor "Red Hat", unknown product
0x000c rev 0x00: apic 0 int 10
pci3 at ppb2 bus 3
ix2 at pci3 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
00:12:c0:88:07:b6
ppb3 at pci0 dev 28 function 3 vendor "Red Hat", unknown product
0x000c rev 0x00: apic 0 int 10
pci4 at ppb3 bus 4
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci5 at ppb4 bus 5
ppb5 at pci5 dev 1 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci6 at ppb5 bus 6
virtio0 at pci6 dev 5 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio0: qsize 128
scsibus1 at vioscsi0: 255 targets
sd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
sd0: 12288MB, 512 bytes/sector, 25165824 sectors, thin
virtio0: msix shared
virtio1 at pci6 dev 18 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address 3a:28:3d:f6:05:45
virtio1: msix shared
ppb6 at pci5 dev 2 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci7 at ppb6 bus 7
ppb7 at pci5 dev 3 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci8 at ppb7 bus 8
ppb8 at pci5 dev 4 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci9 at ppb8 bus 9
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
ahci0: port 1: 1.5Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 1 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 10
iic0 at ichiic0
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (cdebca84b665f93b.a) swap on sd0b dump on sd0b
--
Kindest regards,
Tom Smyth.
