On Mon, Jun 22, 2026 at 7:30 PM Ryan Libby <[email protected]> wrote:
>
> On Mon, Jun 22, 2026 at 4:08 PM Alan Somers <[email protected]> wrote:
> >
> > On Wed, Jun 17, 2026 at 10:18 PM Ryan Libby <[email protected]> wrote:
> > >
> > > The branch main has been updated by rlibby:
> > >
> > > URL:
> > > https://cgit.FreeBSD.org/src/commit/?id=08180f1b613b6fe000135efa28b7c3293cbf1683
> > >
> > > commit 08180f1b613b6fe000135efa28b7c3293cbf1683
> > > Author: Ryan Libby <[email protected]>
> > > AuthorDate: 2026-06-18 03:52:53 +0000
> > > Commit: Ryan Libby <[email protected]>
> > > CommitDate: 2026-06-18 04:05:43 +0000
> > >
> > > witness: actually set read-only tunables in time for witness_startup
> > >
> > > SYSCTL_XXX with CTLFLAG_RDTUN and without CTLFLAG_NOFETCH should not
> > > be
> > > used for values that are needed before SI_SUB_KLD. Otherwise they are
> > > tuned after they are needed. Set CTLFLAG_RDTUN | CTLFLAG_NOFETCH for
> > > the debug.witness.witness_count and debug.witness.skipspin sysctls and
> > > add separate tunables for them, which run at SI_SUB_TUNABLES time,
> > > i.e.,
> > > in time for witness_startup.
> > >
> > > Reviewed by: kib, markj
> > > Sponsored by: Dell Inc.
> > > Differential Revision: https://reviews.freebsd.org/D57613
> >
> > My dev VM instapanics on boot, and I've bisected it to this change.
> > Apparently it doesn't affect everybody, or it would've been reported
> > already. So I'm not asking you to revert it. But could you please
> > take a look at the problem?
> >
> > Stack trace:
> >
> > ---<<BOOT>>---
> > Copyright (c) 1992-2026 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> > The Regents of the University of California. All rights reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 16.0-CURRENT #25 n286717-a9e12d3ec8bd: Mon Jun 22 16:50:06 MDT 2026
> >
> > [email protected]:/usr/obj/usr/home/somers/src/freebsd.org/src/main/amd64.amd64/sys/GENERIC
> > amd64
> > FreeBSD clang version 21.1.8 (https://github.com/llvm/llvm-project.git
> > llvmorg-21.1.8-0-g2078da43e25a)
> > WARNING: WITNESS option enabled, expect reduced performance.
> > SRAT: Ignoring memory at addr 0xff000000
> > panic: physical address 0x4af918000 not covered by the DMAP
> > cpuid = 0
> > time = 1
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame
> > 0xffffffff8293bd60
> > vpanic() at vpanic+0x149/frame 0xffffffff8293be90
> > panic() at panic+0x43/frame 0xffffffff8293bef0
> > pmap_map() at pmap_map+0x2c/frame 0xffffffff8293bf00
> > vm_page_startup() at vm_page_startup+0x71c/frame 0xffffffff8293bf90
> > vm_mem_init() at vm_mem_init+0x1a/frame 0xffffffff8293bfa0
> > mi_startup() at mi_startup+0x1ed/frame 0xffffffff8293bff0
> > KDB: enter: panic
> > [ thread pid 0 tid 0 ]
> > Stopped at kdb_enter+0x33: movq $0,0x15ad202(%rip)
> >
> > /boot/loader.conf contents:
> >
> > kern.boottrace.enabled=1
> > debug.kasan.panic_on_violation=0
> > debug.kmsan.panic_on_violation=0
> > debug.witness.witness_count=49152
>
> Thanks for the report.
>
> I'm able to repro with
> set debug.witness.witness_count=49152
> at the loader prompt.
>
> The WITNESS_COUNT default is 1536. There is an allocation that is
> O(n^2) of witness_count. The setting of 49152 should end up
> attempting to allocate over 2.25 GiB... but actually it will overflow
> a 32-bit int to become negative, leading to us trying to pmap_map
> memory somewhere after the end of the biggest phys segment.
>
> I think a minimum fix would change the types to prevent overflow.
> Actually we also don't check that we aren't just exceeding the size of
> the segment, so presumably we could also hit another panic in just
> trying to allocate too much.
>
> This was a sort of latent issue: before your setting of witness_count
> was just being ignored.
>
> Ryan
asomers, I just pushed: 2407abc9fb3b ("witness: harden tunables for
large settings")
That should avoid the panic with your settings. Thanks to kib's
suggestion, if you set boot verbose you should be able to see a log
line in /var/log/messages about how much startup memory witness wants.
In any case you should also see log lines if it can't be satisfied and
is automatically reduced. Your setting of
debug.witness.witness_count=49152 should result in a request of over
2.25 GiB of startup allocations because witness uses an O(n^2)
adjacency matrix. I'm curious whether you are testing something with
that many unique lock names and if so whether witness is solving your
use case.
I wonder if a different graph representation might be better for that
use case. I assume witness is designed with an adjacency matrix for
speed. Then again I'm unsure if this has been examined from a
performance perspective, and specifically I wonder about the pointer
indirection vs just calculating indices.
Ryan
