Greetings,
I am also seeing increased latency and jitter using an Octeon system -
Edgerouter 6p. It's the same issue for both OpenBSD 7.6 and 7.7. I don't
see it on a amd64 machine with the same configuration.
Attached is a graph showing on the left ping data collected with traffic
going through a PC-Engines APU1 board with OpenBSD 7.7. To the right is
traffic going through an Edgerouter 6p with OpenBSD 7.6 . It's the same
for OpenBSD 7.7.
In my case the config is with OpenBGPD on a vlan. This data is through
NAT but it's the same without NAT and it's also the same using PPPoE
rather than BGPD.
From what I've read on the list it has been observed for Octeon, and
also occasionally on other architectures. As my data shows there are
cases where the amd64 architecture performs as expected.
Is it time for a bug report?
Here is some further info on my system, though note it's 7.6 at the
moment, which I put on temporarily to check if it is unique to version
7.7, which it isn't.
> pfctl -si
Status: Enabled for 0 days 01:20:57 Debug: err
State Table Total Rate
current entries 10
half-open tcp 0
searches 67352 13.9/s
inserts 1262 0.3/s
removals 1252 0.3/s
Counters
match 15580 3.2/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 4 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 22 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 1 0.0/s
synproxy 0 0.0/s
translate 0 0.0/s
no-route 0 0.0/s
> netstat -m
1551 mbufs in use:
1538 mbufs allocated to data
5 mbufs allocated to packet headers
8 mbufs allocated to socket names and addresses
1536/1712 mbuf 2048 byte clusters in use (current/peak)
0/0 mbuf 2112 byte clusters in use (current/peak)
0/40 mbuf 4096 byte clusters in use (current/peak)
0/24 mbuf 8192 byte clusters in use (current/peak)
0/0 mbuf 9216 byte clusters in use (current/peak)
0/0 mbuf 12288 byte clusters in use (current/peak)
0/0 mbuf 16384 byte clusters in use (current/peak)
0/8 mbuf 65536 byte clusters in use (current/peak)
4672/4720/131072 Kbytes allocated to network (current/peak/max)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
0 defrag mbuf allocation
1001 prepend mbuf allocation
0 pullup mbuf allocation
0 pullup memory copy
0 pulldown mbuf allocation
0 pulldown memory copy
> systat vm
2 users Load 0.00 0.00 0.00 15:14:23
memory totals (in KB) PAGING SWAPPING
Interrupts
real virtual free in out in out 576
total
Active 95280 95280 630160 ops soft
All 380976 380976 841072 pages 528
clock
3
cnmac1
Proc:r d s w Csw Trp Sys Int Sof Flt forks 6
cnmac2
58 57 7 63 576 100 13 fkppw
com0
fksvm
octmmc0
0.0%Int 0.0%Spn 0.1%Sys 0.0%Usr 99.9%Idle pwait 39 ipi
| | | | | | | | | | | relck
rlkok
noram
Namei Sys-cache Proc-cache No-cache 1 ndcpy
Calls hits % hits % miss % fltcp
10 10 100 2 zfod
cow
Disks sd0 2106 fmin
seeks 2808 ftarg
xfers itarg
speed 2 wired
sec pdfre
pdscn
pzidl 14
IPKTS
8 kmape 12
OPKTS
> systat mbufs
cnmac4 2 users Load 0.00 0.00 0.00 15:15:15
IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
System mbufs 0 256 1551 27
mcl2k 2048 1536 214
mcl4k 4096 0 5
mcl8k 8192 0 3
mcl64k 65536 0 1
lo0
cnmac0
cnmac1
cnmac2
cnmac3
cnmac4
cnmac5
enc0
vether0
vlan100
dmesg:
root@hrr67:~$ dmesg
[ using 773576 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2024 OpenBSD. All rights reserved.
https://www.OpenBSD.org
OpenBSD 7.6 (GENERIC.MP) #233: Mon Sep 30 09:41:48 MDT 2024
dera...@octeon.openbsd.org:/usr/src/sys/arch/octeon/compile/GENERIC.MP
real mem = 1073741824 (1024MB)
avail mem = 1035321344 (987MB)
random: good seed from bootblocks
mainbus0 at root: board 20300 rev 1.23, model cavium,ubnt_e300
cpu0 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
rev 0.0
cpu0: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
cpu1 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
rev 0.0
cpu1: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
cpu2 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
rev 0.0
cpu2: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
cpu3 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
rev 0.0
cpu3: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
clock0 at mainbus0: int 5
octcrypto0 at mainbus0
iobus0 at mainbus0
simplebus0 at iobus0: "soc"
"bootbus" at simplebus0 not configured
octciu0 at simplebus0
octcib0 at simplebus0: max-bits 23
octcib1 at simplebus0: max-bits 12
octcib2 at simplebus0: max-bits 6
octcib3 at simplebus0: max-bits 15
octcib4 at simplebus0: max-bits 4
octcib5 at simplebus0: max-bits 11
octcib6 at simplebus0: max-bits 11
octgpio0 at simplebus0: 20 pins, xbit 16
octsmi0 at simplebus0
octsmi1 at simplebus0
octpip0 at simplebus0
octgmx0 at octpip0 interface 0
cnmac0 at octgmx0: port 0 SGMII, address 18:e8:29:ba:19:1b
ukphy0 at cnmac0 phy 4: Generic IEEE 802.3u media interface, rev. 2: OUI
0x0001c1, model 0x000c
cnmac1 at octgmx0: port 1 SGMII, address 18:e8:29:ba:19:1c
ukphy1 at cnmac1 phy 5: Generic IEEE 802.3u media interface, rev. 2: OUI
0x0001c1, model 0x000c
cnmac2 at octgmx0: port 2 SGMII, address 18:e8:29:ba:19:1d
ukphy2 at cnmac2 phy 6: Generic IEEE 802.3u media interface, rev. 2: OUI
0x0001c1, model 0x000c
cnmac3 at octgmx0: port 3 SGMII, address 18:e8:29:ba:19:1e
ukphy3 at cnmac3 phy 7: Generic IEEE 802.3u media interface, rev. 2: OUI
0x0001c1, model 0x000c
octgmx1 at octpip0 interface 1
cnmac4 at octgmx1: port 16 SGMII, address 18:e8:29:ba:19:1f
ukphy4 at cnmac4 phy 8: Generic IEEE 802.3u media interface, rev. 0: OUI
0x0001c1, model 0x0027
cnmac5 at octgmx1: port 17 SGMII, address 18:e8:29:ba:19:20
ukphy5 at cnmac5 phy 9: Generic IEEE 802.3u media interface, rev. 0: OUI
0x0001c1, model 0x0027
octsctl0 at simplebus0: disabled
octxctl0 at simplebus0: DWC3 rev 0x250a
xhci0 at octxctl0, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev
3.00/1.00 addr 1
octxctl1 at simplebus0: DWC3 rev 0x250a
xhci1 at octxctl1, xHCI 1.0
usb1 at xhci1: USB revision 3.0
uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root hub" rev
3.00/1.00 addr 1
"i2c" at simplebus0 not configured
"i2c" at simplebus0 not configured
com0 at simplebus0: ns16550a, 64 byte fifo
com0: console
com1 at simplebus0: ns16550a, 64 byte fifo
com1: probed fifo depth: 0 bytes
octmmc0 at simplebus0
sdmmc0 at octmmc0: 8-bit, mmc high-speed
"spi" at simplebus0 not configured
"ocla0" at simplebus0 not configured
"dma-engine" at simplebus0 not configured
"dma-engine" at simplebus0 not configured
octrng0 at iobus0 base 0x1400000000000 irq 0
octpcie0 at iobus0: 3 ports
octpcie0 port 0: link timeout
octpcie0 port 1: reset timeout
octpcie0 port 2: reset timeout
scsibus0 at sdmmc0: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <Kingston, MMC4GB, 0000> removable
sd0: 3728MB, 512 bytes/sector, 7634944 sectors
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (285e489508f0f02b.a) swap on sd0b dump on sd0b
