> On 30 Jun 2025, at 05:00, Mark de Vries <m...@hubs.net.uk> wrote:
>
> I applied the patch and compiled the kernel but it make no difference.
Cool, good to know. I guess I should get a test setup going.
>
> If you want me to do further testing I am now set up to do that a bit more
> quickly.
>
> Mark.
>
>
> On 6/24/25 07:54, David Gwynne wrote:
>> On 24/06/2025 16:39, Mark de Vries wrote:
>>> Will be the first time I apply a patch and compile BSD from source but will
>>> have a go. I suppose this is on the 'current' tree?
>> it should apply to stable too. this code hasn't changed in a while.
>>>
>>> Mark.
>>>
>>>
>>>
>>> On 6/24/25 07:19, David Gwynne wrote:
>>>> On Mon, Jun 23, 2025 at 03:23:27PM +0100, Mark de Vries wrote:
>>>>> Greetings,
>>>>>
>>>>> I am also seeing increased latency and jitter using an Octeon system -
>>>>> Edgerouter 6p. It's the same issue for both OpenBSD 7.6 and 7.7. I don't
>>>>> see
>>>>> it on a amd64 machine with the same configuration.
>>>>>
>>>>> Attached is a graph showing on the left ping data collected with traffic
>>>>> going through a PC-Engines APU1 board with OpenBSD 7.7. To the right is
>>>>> traffic going through an Edgerouter 6p with OpenBSD 7.6 . It's the same
>>>>> for
>>>>> OpenBSD 7.7.
>>>>>
>>>>> In my case the config is with OpenBGPD on a vlan. This data is through NAT
>>>>> but it's the same without NAT and it's also the same using PPPoE rather
>>>>> than
>>>>> BGPD.
>>>>>
>>>>> From what I've read on the list it has been observed for Octeon, and also
>>>>> occasionally on other architectures. As my data shows there are cases
>>>>> where
>>>>> the amd64 architecture performs as expected.
>>>>>
>>>>> Is it time for a bug report?
>>>>>
>>>>> Here is some further info on my system, though note it's 7.6 at the
>>>>> moment,
>>>>> which I put on temporarily to check if it is unique to version 7.7, which
>>>>> it
>>>>> isn't.
>>>>>
>>>>>
>>>>>> pfctl -si
>>>>>
>>>>> Status: Enabled for 0 days 01:20:57 Debug: err
>>>>>
>>>>> State Table Total Rate
>>>>> current entries 10
>>>>> half-open tcp 0
>>>>> searches 67352 13.9/s
>>>>> inserts 1262 0.3/s
>>>>> removals 1252 0.3/s
>>>>> Counters
>>>>> match 15580 3.2/s
>>>>> bad-offset 0 0.0/s
>>>>> fragment 0 0.0/s
>>>>> short 0 0.0/s
>>>>> normalize 4 0.0/s
>>>>> memory 0 0.0/s
>>>>> bad-timestamp 0 0.0/s
>>>>> congestion 0 0.0/s
>>>>> ip-option 0 0.0/s
>>>>> proto-cksum 0 0.0/s
>>>>> state-mismatch 22 0.0/s
>>>>> state-insert 0 0.0/s
>>>>> state-limit 0 0.0/s
>>>>> src-limit 1 0.0/s
>>>>> synproxy 0 0.0/s
>>>>> translate 0 0.0/s
>>>>> no-route 0 0.0/s
>>>>>
>>>>>> netstat -m
>>>>> 1551 mbufs in use:
>>>>> 1538 mbufs allocated to data
>>>>> 5 mbufs allocated to packet headers
>>>>> 8 mbufs allocated to socket names and addresses
>>>>> 1536/1712 mbuf 2048 byte clusters in use (current/peak)
>>>>> 0/0 mbuf 2112 byte clusters in use (current/peak)
>>>>> 0/40 mbuf 4096 byte clusters in use (current/peak)
>>>>> 0/24 mbuf 8192 byte clusters in use (current/peak)
>>>>> 0/0 mbuf 9216 byte clusters in use (current/peak)
>>>>> 0/0 mbuf 12288 byte clusters in use (current/peak)
>>>>> 0/0 mbuf 16384 byte clusters in use (current/peak)
>>>>> 0/8 mbuf 65536 byte clusters in use (current/peak)
>>>>> 4672/4720/131072 Kbytes allocated to network (current/peak/max)
>>>>> 0 requests for memory denied
>>>>> 0 requests for memory delayed
>>>>> 0 calls to protocol drain routines
>>>>> 0 defrag mbuf allocation
>>>>> 1001 prepend mbuf allocation
>>>>> 0 pullup mbuf allocation
>>>>> 0 pullup memory copy
>>>>> 0 pulldown mbuf allocation
>>>>> 0 pulldown memory copy
>>>>>
>>>>>> systat vm
>>>>>
>>>>>
>>>>> 2 users Load 0.00 0.00 0.00 15:14:23
>>>>>
>>>>> memory totals (in KB) PAGING SWAPPING Interrupts
>>>>> real virtual free in out in out 576
>>>>> total
>>>>> Active 95280 95280 630160 ops soft
>>>>> All 380976 380976 841072 pages 528
>>>>> clock
>>>>> 3
>>>>> cnmac1
>>>>> Proc:r d s w Csw Trp Sys Int Sof Flt forks 6
>>>>> cnmac2
>>>>> 58 57 7 63 576 100 13 fkppw com0
>>>>> fksvm octmmc0
>>>>> 0.0%Int 0.0%Spn 0.1%Sys 0.0%Usr 99.9%Idle pwait 39 ipi
>>>>> | | | | | | | | | | | relck
>>>>> rlkok
>>>>> noram
>>>>> Namei Sys-cache Proc-cache No-cache 1 ndcpy
>>>>> Calls hits % hits % miss % fltcp
>>>>> 10 10 100 2 zfod
>>>>> cow
>>>>> Disks sd0 2106 fmin
>>>>> seeks 2808 ftarg
>>>>> xfers itarg
>>>>> speed 2 wired
>>>>> sec pdfre
>>>>> pdscn
>>>>> pzidl 14
>>>>> IPKTS
>>>>> 8 kmape 12
>>>>> OPKTS
>>>>>
>>>>>> systat mbufs
>>>>>
>>>>>
>>>>> cnmac4 2 users Load 0.00 0.00 0.00 15:15:15
>>>>>
>>>>> IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
>>>>> System mbufs 0 256 1551 27
>>>>> mcl2k 2048 1536 214
>>>>> mcl4k 4096 0 5
>>>>> mcl8k 8192 0 3
>>>>> mcl64k 65536 0 1
>>>>> lo0
>>>>> cnmac0
>>>>> cnmac1
>>>>> cnmac2
>>>>> cnmac3
>>>>> cnmac4
>>>>> cnmac5
>>>>> enc0
>>>>> vether0
>>>>> vlan100
>>>>>
>>>>> dmesg:
>>>>> root@hrr67:~$ dmesg
>>>>> [ using 773576 bytes of bsd ELF symbol table ]
>>>>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>>>>> The Regents of the University of California. All rights
>>>>> reserved.
>>>>> Copyright (c) 1995-2024 OpenBSD. All rights reserved.
>>>>> https://www.OpenBSD.org
>>>>>
>>>>> OpenBSD 7.6 (GENERIC.MP) #233: Mon Sep 30 09:41:48 MDT 2024
>>>>> dera...@octeon.openbsd.org:/usr/src/sys/arch/octeon/compile/GENERIC.MP
>>>>> real mem = 1073741824 (1024MB)
>>>>> avail mem = 1035321344 (987MB)
>>>>> random: good seed from bootblocks
>>>>> mainbus0 at root: board 20300 rev 1.23, model cavium,ubnt_e300
>>>>> cpu0 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
>>>>> rev
>>>>> 0.0
>>>>> cpu0: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>> cpu1 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
>>>>> rev
>>>>> 0.0
>>>>> cpu1: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>> cpu2 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
>>>>> rev
>>>>> 0.0
>>>>> cpu2: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>> cpu3 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx FPU
>>>>> rev
>>>>> 0.0
>>>>> cpu3: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>> clock0 at mainbus0: int 5
>>>>> octcrypto0 at mainbus0
>>>>> iobus0 at mainbus0
>>>>> simplebus0 at iobus0: "soc"
>>>>> "bootbus" at simplebus0 not configured
>>>>> octciu0 at simplebus0
>>>>> octcib0 at simplebus0: max-bits 23
>>>>> octcib1 at simplebus0: max-bits 12
>>>>> octcib2 at simplebus0: max-bits 6
>>>>> octcib3 at simplebus0: max-bits 15
>>>>> octcib4 at simplebus0: max-bits 4
>>>>> octcib5 at simplebus0: max-bits 11
>>>>> octcib6 at simplebus0: max-bits 11
>>>>> octgpio0 at simplebus0: 20 pins, xbit 16
>>>>> octsmi0 at simplebus0
>>>>> octsmi1 at simplebus0
>>>>> octpip0 at simplebus0
>>>>> octgmx0 at octpip0 interface 0
>>>>> cnmac0 at octgmx0: port 0 SGMII, address 18:e8:29:ba:19:1b
>>>>> ukphy0 at cnmac0 phy 4: Generic IEEE 802.3u media interface, rev. 2: OUI
>>>>> 0x0001c1, model 0x000c
>>>>> cnmac1 at octgmx0: port 1 SGMII, address 18:e8:29:ba:19:1c
>>>>> ukphy1 at cnmac1 phy 5: Generic IEEE 802.3u media interface, rev. 2: OUI
>>>>> 0x0001c1, model 0x000c
>>>>> cnmac2 at octgmx0: port 2 SGMII, address 18:e8:29:ba:19:1d
>>>>> ukphy2 at cnmac2 phy 6: Generic IEEE 802.3u media interface, rev. 2: OUI
>>>>> 0x0001c1, model 0x000c
>>>>> cnmac3 at octgmx0: port 3 SGMII, address 18:e8:29:ba:19:1e
>>>>> ukphy3 at cnmac3 phy 7: Generic IEEE 802.3u media interface, rev. 2: OUI
>>>>> 0x0001c1, model 0x000c
>>>>> octgmx1 at octpip0 interface 1
>>>>> cnmac4 at octgmx1: port 16 SGMII, address 18:e8:29:ba:19:1f
>>>>> ukphy4 at cnmac4 phy 8: Generic IEEE 802.3u media interface, rev. 0: OUI
>>>>> 0x0001c1, model 0x0027
>>>>> cnmac5 at octgmx1: port 17 SGMII, address 18:e8:29:ba:19:20
>>>>> ukphy5 at cnmac5 phy 9: Generic IEEE 802.3u media interface, rev. 0: OUI
>>>>> 0x0001c1, model 0x0027
>>>>> octsctl0 at simplebus0: disabled
>>>>> octxctl0 at simplebus0: DWC3 rev 0x250a
>>>>> xhci0 at octxctl0, xHCI 1.0
>>>>> usb0 at xhci0: USB revision 3.0
>>>>> uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev
>>>>> 3.00/1.00 addr 1
>>>>> octxctl1 at simplebus0: DWC3 rev 0x250a
>>>>> xhci1 at octxctl1, xHCI 1.0
>>>>> usb1 at xhci1: USB revision 3.0
>>>>> uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root hub" rev
>>>>> 3.00/1.00 addr 1
>>>>> "i2c" at simplebus0 not configured
>>>>> "i2c" at simplebus0 not configured
>>>>> com0 at simplebus0: ns16550a, 64 byte fifo
>>>>> com0: console
>>>>> com1 at simplebus0: ns16550a, 64 byte fifo
>>>>> com1: probed fifo depth: 0 bytes
>>>>> octmmc0 at simplebus0
>>>>> sdmmc0 at octmmc0: 8-bit, mmc high-speed
>>>>> "spi" at simplebus0 not configured
>>>>> "ocla0" at simplebus0 not configured
>>>>> "dma-engine" at simplebus0 not configured
>>>>> "dma-engine" at simplebus0 not configured
>>>>> octrng0 at iobus0 base 0x1400000000000 irq 0
>>>>> octpcie0 at iobus0: 3 ports
>>>>> octpcie0 port 0: link timeout
>>>>> octpcie0 port 1: reset timeout
>>>>> octpcie0 port 2: reset timeout
>>>>> scsibus0 at sdmmc0: 2 targets, initiator 0
>>>>> sd0 at scsibus0 targ 1 lun 0: <Kingston, MMC4GB, 0000> removable
>>>>> sd0: 3728MB, 512 bytes/sector, 7634944 sectors
>>>>> vscsi0 at root
>>>>> scsibus1 at vscsi0: 256 targets
>>>>> softraid0 at root
>>>>> scsibus2 at softraid0: 256 targets
>>>>> root on sd0a (285e489508f0f02b.a) swap on sd0b dump on sd0b
>>>>
>>>> can you try this?
>>>>
>>>> Index: uipc_socket.c
>>>> ===================================================================
>>>> RCS file: /cvs/src/sys/kern/uipc_socket.c,v
>>>> diff -u -p -r1.344 uipc_socket.c
>>>> --- uipc_socket.c 31 Oct 2024 12:51:55 -0000 1.344
>>>> +++ uipc_socket.c 1 Nov 2024 23:53:25 -0000
>>>> @@ -787,48 +787,36 @@ m_getuio(struct mbuf **mp, int atomic, l
>>>> {
>>>> struct mbuf *m, *top = NULL;
>>>> struct mbuf **nextp = ⊤
>>>> - u_long len, mlen;
>>>> - size_t resid = uio->uio_resid;
>>>> + u_long len, mlen, alen;
>>>> + int align = atomic ? roundup(max_hdr, sizeof(long)) : 0;
>>>> int error;
>>>> - do {
>>>> - if (top == NULL) {
>>>> - MGETHDR(m, M_WAIT, MT_DATA);
>>>> - mlen = MHLEN;
>>>> - } else {
>>>> - MGET(m, M_WAIT, MT_DATA);
>>>> - mlen = MLEN;
>>>> - }
>>>> + m = m_gethdr(M_WAIT, MT_DATA);
>>>> + mlen = MHLEN;
>>>> +
>>>> + for (;;) {
>>>> /* chain mbuf together */
>>>> *nextp = m;
>>>> nextp = &m->m_next;
>>>> - resid = ulmin(resid, space);
>>>> - if (resid >= MINCLSIZE) {
>>>> - MCLGETL(m, M_NOWAIT, ulmin(resid, MAXMCLBYTES));
>>>> - if ((m->m_flags & M_EXT) == 0)
>>>> + /* How much data we want to put in this mbuf? */
>>>> + len = ulmin(uio->uio_resid, space);
>>>> + /* How much space are we allocating for that data? */
>>>> + alen = align + len;
>>>> + if (alen > mlen) {
>>>> + MCLGETL(m, M_NOWAIT, ulmin(alen, MAXMCLBYTES));
>>>> + if (!ISSET(m->m_flags, M_EXT) && alen > MCLBYTES)
>>>> MCLGETL(m, M_NOWAIT, MCLBYTES);
>>>> - if ((m->m_flags & M_EXT) == 0)
>>>> - goto nopages;
>>>> - mlen = m->m_ext.ext_size;
>>>> - len = ulmin(mlen, resid);
>>>> - /*
>>>> - * For datagram protocols, leave room
>>>> - * for protocol headers in first mbuf.
>>>> - */
>>>> - if (atomic && m == top && len < mlen - max_hdr)
>>>> - m->m_data += max_hdr;
>>>> - } else {
>>>> -nopages:
>>>> - len = ulmin(mlen, resid);
>>>> - /*
>>>> - * For datagram protocols, leave room
>>>> - * for protocol headers in first mbuf.
>>>> - */
>>>> - if (atomic && m == top && len < mlen - max_hdr)
>>>> - m_align(m, len);
>>>> + if (ISSET(m->m_flags, M_EXT))
>>>> + mlen = m->m_ext.ext_size;
>>>> }
>>>> + /* Avoid pain from a stupid max_hdr value */
>>>> + if (align < mlen)
>>>> + m->m_data += align;
>>>> +
>>>> + /* How much data can we put in this mbuf? */
>>>> + len = ulmin(mlen, len);
>>>> error = uiomove(mtod(m, caddr_t), len, uio);
>>>> if (error) {
>>>> m_freem(top);
>>>> @@ -836,13 +824,19 @@ nopages:
>>>> }
>>>> /* adjust counters */
>>>> - resid = uio->uio_resid;
>>>> space -= len;
>>>> m->m_len = len;
>>>> top->m_pkthdr.len += len;
>>>> /* Is there more space and more data? */
>>>> - } while (space > 0 && resid > 0);
>>>> + if (space == 0 || uio->uio_resid == 0)
>>>> + break;
>>>> +
>>>> + align = 0;
>>>> +
>>>> + m = m_get(M_WAIT, MT_DATA);
>>>> + mlen = MLEN;
>>>> + }
>>>> *mp = top;
>>>> return 0;
>>>>
>>>
>>>
>
>
