Re: Ping Spikes

David Gwynne Mon, 23 Jun 2025 23:55:56 -0700



On 24/06/2025 16:39, Mark de Vries wrote:

Will be the first time I apply a patch and compile BSD from source butwill have a go. I suppose this is on the 'current' tree?


it should apply to stable too. this code hasn't changed in a while.


Mark.



On 6/24/25 07:19, David Gwynne wrote:

On Mon, Jun 23, 2025 at 03:23:27PM +0100, Mark de Vries wrote:

Greetings,

I am also seeing increased latency and jitter using an Octeon system -

Edgerouter 6p. It's the same issue for both OpenBSD 7.6 and 7.7. Idon't see

it on a amd64 machine with the same configuration.

Attached is a graph showing on the left ping data collected withtraffic

going through a PC-Engines APU1 board with OpenBSD 7.7. To the right is

traffic going through an Edgerouter 6p with OpenBSD 7.6 . It's thesame for

OpenBSD 7.7.

In my case the config is with OpenBGPD on a vlan. This data isthrough NATbut it's the same without NAT and it's also the same using PPPoErather than

BGPD.

From what I've read on the list it has been observed for Octeon,and alsooccasionally on other architectures. As my data shows there arecases where

the amd64 architecture performs as expected.

Is it time for a bug report?

Here is some further info on my system, though note it's 7.6 at themoment,which I put on temporarily to check if it is unique to version 7.7,which it

isn't.

pfctl -si


Status: Enabled for 0 days 01:20:57              Debug: err

State Table                          Total             Rate
   current entries                       10
   half-open tcp                          0
   searches                           67352           13.9/s
   inserts                             1262            0.3/s
   removals                            1252            0.3/s
Counters
   match                              15580            3.2/s
   bad-offset                             0            0.0/s
   fragment                               0            0.0/s
   short                                  0            0.0/s
   normalize                              4            0.0/s
   memory                                 0            0.0/s
   bad-timestamp                          0            0.0/s
   congestion                             0            0.0/s
   ip-option                              0            0.0/s
   proto-cksum                            0            0.0/s
   state-mismatch                        22            0.0/s
   state-insert                           0            0.0/s
   state-limit                            0            0.0/s
   src-limit                              1            0.0/s
   synproxy                               0            0.0/s
   translate                              0            0.0/s
   no-route                               0            0.0/s

netstat -m

1551 mbufs in use:
         1538 mbufs allocated to data
         5 mbufs allocated to packet headers
         8 mbufs allocated to socket names and addresses
1536/1712 mbuf 2048 byte clusters in use (current/peak)
0/0 mbuf 2112 byte clusters in use (current/peak)
0/40 mbuf 4096 byte clusters in use (current/peak)
0/24 mbuf 8192 byte clusters in use (current/peak)
0/0 mbuf 9216 byte clusters in use (current/peak)
0/0 mbuf 12288 byte clusters in use (current/peak)
0/0 mbuf 16384 byte clusters in use (current/peak)
0/8 mbuf 65536 byte clusters in use (current/peak)
4672/4720/131072 Kbytes allocated to network (current/peak/max)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
0 defrag mbuf allocation
1001 prepend mbuf allocation
0 pullup mbuf allocation
0 pullup memory copy
0 pulldown mbuf allocation
0 pulldown memory copy

systat vm



   2 users Load 0.00 0.00 0.00                        15:14:23

memory totals (in KB) PAGING SWAPPINGInterrupts

            real   virtual     free           in  out   in out      576
total

Active 95280 95280 630160ops soft

All      380976    380976   841072 pages                          528
clock

3

cnmac1
Proc:r  d  s  w    Csw   Trp   Sys   Int   Sof  Flt forks       6
cnmac2
           58        57     7    63   576   100   13 fkppw com0
fksvm octmmc0
    0.0%Int   0.0%Spn   0.1%Sys   0.0%Usr  99.9%Idle pwait      39 ipi
|    |    |    |    |    |    |    |    |    |    | relck
rlkok
noram
Namei         Sys-cache    Proc-cache    No-cache       1 ndcpy
     Calls     hits    %    hits     %    miss   % fltcp
        10       10  100                                 2 zfod
                                                           cow
Disks   sd0                                          2106 fmin
seeks                                                2808 ftarg
xfers itarg
speed                                                   2 wired
   sec pdfre
pdscn
pzidl      14
IPKTS

8kmape 12

OPKTS

systat mbufs



cnmac4   2 users Load 0.00 0.00 0.00 15:15:15

IFACE            RING      LIVELOCKS  SIZE ALIVE   LWM   HWM CWM
System           mbufs             0   256  1551          27
                  mcl2k                2048  1536         214
                  mcl4k                4096     0           5
                  mcl8k                8192     0           3
                  mcl64k              65536     0           1
lo0
cnmac0
cnmac1
cnmac2
cnmac3
cnmac4
cnmac5
enc0
vether0
vlan100

dmesg:
root@hrr67:~$ dmesg
[ using 773576 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993

The Regents of the University of California. All rightsreserved.

Copyright (c) 1995-2024 OpenBSD. All rights reserved.
https://www.OpenBSD.org

OpenBSD 7.6 (GENERIC.MP) #233: Mon Sep 30 09:41:48 MDT 2024
dera...@octeon.openbsd.org:/usr/src/sys/arch/octeon/compile/GENERIC.MP
real mem = 1073741824 (1024MB)
avail mem = 1035321344 (987MB)
random: good seed from bootblocks
mainbus0 at root: board 20300 rev 1.23, model cavium,ubnt_e300

cpu0 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xxFPU rev

0.0
cpu0: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way

cpu1 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xxFPU rev

0.0
cpu1: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way

cpu2 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xxFPU rev

0.0
cpu2: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way

cpu3 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xxFPU rev

0.0
cpu3: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
clock0 at mainbus0: int 5
octcrypto0 at mainbus0
iobus0 at mainbus0
simplebus0 at iobus0: "soc"
"bootbus" at simplebus0 not configured
octciu0 at simplebus0
octcib0 at simplebus0: max-bits 23
octcib1 at simplebus0: max-bits 12
octcib2 at simplebus0: max-bits 6
octcib3 at simplebus0: max-bits 15
octcib4 at simplebus0: max-bits 4
octcib5 at simplebus0: max-bits 11
octcib6 at simplebus0: max-bits 11
octgpio0 at simplebus0: 20 pins, xbit 16
octsmi0 at simplebus0
octsmi1 at simplebus0
octpip0 at simplebus0
octgmx0 at octpip0 interface 0
cnmac0 at octgmx0: port 0 SGMII, address 18:e8:29:ba:19:1b

ukphy0 at cnmac0 phy 4: Generic IEEE 802.3u media interface, rev. 2:OUI

0x0001c1, model 0x000c
cnmac1 at octgmx0: port 1 SGMII, address 18:e8:29:ba:19:1c

ukphy1 at cnmac1 phy 5: Generic IEEE 802.3u media interface, rev. 2:OUI

0x0001c1, model 0x000c
cnmac2 at octgmx0: port 2 SGMII, address 18:e8:29:ba:19:1d

ukphy2 at cnmac2 phy 6: Generic IEEE 802.3u media interface, rev. 2:OUI

0x0001c1, model 0x000c
cnmac3 at octgmx0: port 3 SGMII, address 18:e8:29:ba:19:1e

ukphy3 at cnmac3 phy 7: Generic IEEE 802.3u media interface, rev. 2:OUI

0x0001c1, model 0x000c
octgmx1 at octpip0 interface 1
cnmac4 at octgmx1: port 16 SGMII, address 18:e8:29:ba:19:1f

ukphy4 at cnmac4 phy 8: Generic IEEE 802.3u media interface, rev. 0:OUI

0x0001c1, model 0x0027
cnmac5 at octgmx1: port 17 SGMII, address 18:e8:29:ba:19:20

ukphy5 at cnmac5 phy 9: Generic IEEE 802.3u media interface, rev. 0:OUI

0x0001c1, model 0x0027
octsctl0 at simplebus0: disabled
octxctl0 at simplebus0: DWC3 rev 0x250a
xhci0 at octxctl0, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev
3.00/1.00 addr 1
octxctl1 at simplebus0: DWC3 rev 0x250a
xhci1 at octxctl1, xHCI 1.0
usb1 at xhci1: USB revision 3.0
uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root hub" rev
3.00/1.00 addr 1
"i2c" at simplebus0 not configured
"i2c" at simplebus0 not configured
com0 at simplebus0: ns16550a, 64 byte fifo
com0: console
com1 at simplebus0: ns16550a, 64 byte fifo
com1: probed fifo depth: 0 bytes
octmmc0 at simplebus0
sdmmc0 at octmmc0: 8-bit, mmc high-speed
"spi" at simplebus0 not configured
"ocla0" at simplebus0 not configured
"dma-engine" at simplebus0 not configured
"dma-engine" at simplebus0 not configured
octrng0 at iobus0 base 0x1400000000000 irq 0
octpcie0 at iobus0: 3 ports
octpcie0 port 0: link timeout
octpcie0 port 1: reset timeout
octpcie0 port 2: reset timeout
scsibus0 at sdmmc0: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <Kingston, MMC4GB, 0000> removable
sd0: 3728MB, 512 bytes/sector, 7634944 sectors
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (285e489508f0f02b.a) swap on sd0b dump on sd0b


can you try this?

Index: uipc_socket.c
===================================================================
RCS file: /cvs/src/sys/kern/uipc_socket.c,v
diff -u -p -r1.344 uipc_socket.c
--- uipc_socket.c    31 Oct 2024 12:51:55 -0000    1.344
+++ uipc_socket.c    1 Nov 2024 23:53:25 -0000
@@ -787,48 +787,36 @@ m_getuio(struct mbuf **mp, int atomic, l
  {
      struct mbuf *m, *top = NULL;
      struct mbuf **nextp = &top;
-    u_long len, mlen;
-    size_t resid = uio->uio_resid;
+    u_long len, mlen, alen;
+    int align = atomic ? roundup(max_hdr, sizeof(long)) : 0;
      int error;
  -    do {
-        if (top == NULL) {
-            MGETHDR(m, M_WAIT, MT_DATA);
-            mlen = MHLEN;
-        } else {
-            MGET(m, M_WAIT, MT_DATA);
-            mlen = MLEN;
-        }
+    m = m_gethdr(M_WAIT, MT_DATA);
+    mlen = MHLEN;
+
+    for (;;) {
          /* chain mbuf together */
          *nextp = m;
          nextp = &m->m_next;
  -        resid = ulmin(resid, space);
-        if (resid >= MINCLSIZE) {
-            MCLGETL(m, M_NOWAIT, ulmin(resid, MAXMCLBYTES));
-            if ((m->m_flags & M_EXT) == 0)
+        /* How much data we want to put in this mbuf? */
+        len = ulmin(uio->uio_resid, space);
+        /* How much space are we allocating for that data? */
+        alen = align + len;
+        if (alen > mlen) {
+            MCLGETL(m, M_NOWAIT, ulmin(alen, MAXMCLBYTES));
+            if (!ISSET(m->m_flags, M_EXT) && alen > MCLBYTES)
                  MCLGETL(m, M_NOWAIT, MCLBYTES);
-            if ((m->m_flags & M_EXT) == 0)
-                goto nopages;
-            mlen = m->m_ext.ext_size;
-            len = ulmin(mlen, resid);
-            /*
-             * For datagram protocols, leave room
-             * for protocol headers in first mbuf.
-             */
-            if (atomic && m == top && len < mlen - max_hdr)
-                m->m_data += max_hdr;
-        } else {
-nopages:
-            len = ulmin(mlen, resid);
-            /*
-             * For datagram protocols, leave room
-             * for protocol headers in first mbuf.
-             */
-            if (atomic && m == top && len < mlen - max_hdr)
-                m_align(m, len);
+            if (ISSET(m->m_flags, M_EXT))
+                mlen = m->m_ext.ext_size;
          }
  +        /* Avoid pain from a stupid max_hdr value */
+        if (align < mlen)
+            m->m_data += align;
+
+        /* How much data can we put in this mbuf? */
+        len = ulmin(mlen, len);
          error = uiomove(mtod(m, caddr_t), len, uio);
          if (error) {
              m_freem(top);
@@ -836,13 +824,19 @@ nopages:
          }
            /* adjust counters */
-        resid = uio->uio_resid;
          space -= len;
          m->m_len = len;
          top->m_pkthdr.len += len;
            /* Is there more space and more data? */
-    } while (space > 0 && resid > 0);
+        if (space == 0 || uio->uio_resid == 0)
+            break;
+
+        align = 0;
+
+        m = m_get(M_WAIT, MT_DATA);
+        mlen = MLEN;
+    }
        *mp = top;
      return 0;

Re: Ping Spikes

Reply via email to