Re: panic - mutex? - AMD64/7.4 to current
On Sun, Aug 25, 2024 at 11:10:19AM +0200, Martin Pieuchot wrote: > On 24/08/24(Sat) 13:14, Hugh Graham wrote: > > On Sat, Aug 24, 2024 at 09:31:43PM +0200, Martin Pieuchot wrote: > > > On 24/08/24(Sat) 12:09, Hugh Graham wrote: > > > > The machine that slowly received the ports tree crashed upon reboot, > > > > and I have included the traces. It was running 7.5-stable at the > > > > time. I will wait an hour or so, and then restart and repeat > > > > against current. > > > > > > Thanks, could you elaborate how you trigger the corruptions? I'd like > > > to try to reproduce them locally. > > > > To summarise, I first encountered the crash while trying to move > > a large filesystem archive from an old machine that could only > > manage roughly 1MB/s to a relatively new machine over ethernet. > > I then reproduced the crash by setting media 10BaseT on the > > receiving machine, and sending the archive from another new > > machine. I haven't been able to reproduce the crash with either > > local or fast network transfers, but also haven't tested this > > exhaustively. > > > > Subsequently, I have reproduced the crash by setting media 10BaseT > > on my target machine, and sending 25GB of ports and packages via > > pax over netcat. Possibly loops of CVSROOT would work better than > > the large packages. There is a dmesg from the target machine > > deeper in this email thread. > > Hugh, could you please send the exact commands that triggered the crash? > All the details matter! OK. Here is a log from today. There are two common themes in these crashes. 1) This possibly cursed machine that should be flung into hades. 2) A slow network and lots of file writes. This time it took about 35GB. The kernel was current as of yesterday, and I should note the machine previously used sata ssd and also crashed in the same circumstances. /Hugh >> OpenBSD/amd64 BOOTX64 3.65 boot> boot bsd.76 -s booting hd0a:bsd.76: 20010965+4318216+417824+0+1241088 [1546704+128+1398696+1094397]=0x1ca58f8 entry point at 0x1001000 [ using 4040952 bytes of bsd ELF symbol table ] Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2024 OpenBSD. All rights reserved. https://www.OpenBSD.org OpenBSD 7.6-beta (GENERIC.MP) #3: Sat Aug 24 11:35:41 PDT 2024 [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 33551818752 (31997MB) avail mem = 32511336448 (31005MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.3 @ 0xc67ed000 (65 entries) bios0: vendor American Megatrends Inc. version "3405" date 12/13/2023 bios0: ASUS PRIME B550M-A efi0 at bios0: UEFI 2.7 efi0: American Megatrends rev 0x50011 acpi0 at bios0: ACPI 6.2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP IVRS SSDT SSDT SSDT FIDT FPDT MCFG HPET VFCT WPBT TPM2 SSDT CRAT CDIT SSDT SSDT SSDT WSMT APIC SSDT SSDT acpi0: wakeup devices X162(S4) GP17(S4) XHC0(S4) XHC1(S4) PS2K(S3) X161(S4) PTXH(S4) X1_1(S4) X1_2(S4) RLAN(S4) M2_2(S4) acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimcfg0 at acpi0 acpimcfg0: addr 0xf000, bus 0-127 acpihpet0 at acpi0: 14318180 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 5 5600G with Radeon Graphics, 3900.01 MHz, 19-50-00, patch 0a5f cpu0: cpuid 1 edx=178bfbff ecx=76f8320b cpu0: cpuid 6 eax=4 ecx=1 cpu0: cpuid 7.0 ebx=219c97a9 ecx=40068c edx=10 cpu0: cpuid d.1 eax=f cpu0: cpuid 8001 edx=2fd3fbff ecx=75c237ff cpu0: cpuid 8007 edx=6799 cpu0: cpuid 8008 ebx=191ef657 cpu0: cpuid 801F eax=1780f edx=1 cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 100MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: AMD Ryzen 5 5600G with Radeon Graphics, 3900.00 MHz, 19-50-00, patch 0a5f cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: AMD Ryzen 5 5600G with Radeon Graphics, 3900.00 MHz, 19-50-00, patch 0a5f cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: AMD Ryzen 5 5600G with Radeon Graphics, 3900.00 MHz, 19-50-00, patch 0a5f cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 8 (application processor) cpu4: AMD Ryzen 5 5600G with Radeon Graphics, 3900.00 MHz, 19-50-00, patch 0a5f cpu4: smt 0, core 4, package 0 cpu5 at mainbus0: apid 10 (application processor) cpu5: AMD Ryzen 5 5600G with Radeon Graphics, 3900.00 MHz, 19-50-00, patch 0a5f cpu5: smt 0, core 5, package 0 cpu6 at mainbus0: apid 1 (application processor) cpu6: AMD Ryzen 5 5600G with Radeon Graphics, 3900.00 MHz, 19-50-00, patch 0a5f cpu6: smt 1, core 0, package 0
Re: panic - mutex? - AMD64/7.4 to current
On 24/08/24(Sat) 13:14, Hugh Graham wrote: > On Sat, Aug 24, 2024 at 09:31:43PM +0200, Martin Pieuchot wrote: > > On 24/08/24(Sat) 12:09, Hugh Graham wrote: > > > The machine that slowly received the ports tree crashed upon reboot, > > > and I have included the traces. It was running 7.5-stable at the > > > time. I will wait an hour or so, and then restart and repeat > > > against current. > > > > Thanks, could you elaborate how you trigger the corruptions? I'd like > > to try to reproduce them locally. > > To summarise, I first encountered the crash while trying to move > a large filesystem archive from an old machine that could only > manage roughly 1MB/s to a relatively new machine over ethernet. > I then reproduced the crash by setting media 10BaseT on the > receiving machine, and sending the archive from another new > machine. I haven't been able to reproduce the crash with either > local or fast network transfers, but also haven't tested this > exhaustively. > > Subsequently, I have reproduced the crash by setting media 10BaseT > on my target machine, and sending 25GB of ports and packages via > pax over netcat. Possibly loops of CVSROOT would work better than > the large packages. There is a dmesg from the target machine > deeper in this email thread. Hugh, could you please send the exact commands that triggered the crash? All the details matter!
Re: panic - mutex? - AMD64/7.4 to current
On Sat, Aug 24, 2024 at 09:31:43PM +0200, Martin Pieuchot wrote:
> On 24/08/24(Sat) 12:09, Hugh Graham wrote:
> > The machine that slowly received the ports tree crashed upon reboot,
> > and I have included the traces. It was running 7.5-stable at the
> > time. I will wait an hour or so, and then restart and repeat
> > against current.
>
> Thanks, could you elaborate how you trigger the corruptions? I'd like
> to try to reproduce them locally.
To summarise, I first encountered the crash while trying to move
a large filesystem archive from an old machine that could only
manage roughly 1MB/s to a relatively new machine over ethernet.
I then reproduced the crash by setting media 10BaseT on the
receiving machine, and sending the archive from another new
machine. I haven't been able to reproduce the crash with either
local or fast network transfers, but also haven't tested this
exhaustively.
Subsequently, I have reproduced the crash by setting media 10BaseT
on my target machine, and sending 25GB of ports and packages via
pax over netcat. Possibly loops of CVSROOT would work better than
the large packages. There is a dmesg from the target machine
deeper in this email thread.
/Hugh
>
> > # reboot
> > dd: /var/db/host.random: No such file or directory
> > chmod: /var/db/host.random: No such file or directory
> > dd: /var/db/host.random: No such file or directory
> > single user: not running shutdown scripts
> > syncing disks...uvm_fault(0xfd880d4cb420, 0x40018, 0, 1) -> e
> > kernel: page fault trap, code=0
> > Stopped at _rb_remove+0x2e7: cmpl$0,0x18(%rax)
> > TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
> > *101771 59108 0 0x803 00K reboot
> > 316626 12974 0 0x14000 0x42002 sensors
> > _rb_remove(822f5390,fd8555e22ba8,fd854219dc48) at
> > _rb_remove+0x
> > 2e7
> > brelse(fd854219dc48) at brelse+0x103
> > vinvalbuf(fd8555e22b40,1,,80006a61c008,0,ff
> > ff) at vinvalbuf+0x1e9
> > vclean(fd8555e22b40,8,80006a61c008) at vclean+0xec
> > vgonel(fd8555e22b40,80006a61c008) at vgonel+0x55
> > vflush_vnode(fd8555e22b40,80006a64d730) at vflush_vnode+0x106
> > vflush(801ccc00,0,2) at vflush+0x57
> > ffs_flushfiles(801ccc00,2,80006a61c008) at ffs_flushfiles+0x8a
> > ffs_unmount(801ccc00,8,80006a61c008) at ffs_unmount+0x3e
> > dounmount_leaf(801ccc00,8,80006a61c008) at
> > dounmount_leaf+0xbe
> > dounmount(801ccc00,8,80006a61c008) at dounmount+0xec
> > vfs_unmountall() at vfs_unmountall+0x7f
> > vfs_shutdown(80006a61c008) at vfs_shutdown+0x3e
> > boot(0) at boot+0x77
> > end trace frame: 0x80006a64d9c0, count: 0
> > https://www.openbsd.org/ddb.html describes the minimum info required in bug
> > reports. Insufficient info makes it difficult to find and fix bugs.
> > ddb{0}> show uvm
> > Current UVM status:
> > pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
> > 7940097 VM pages: 98 active, 1215 inactive, 1 wired, 4713505 free (987509
> > zer
> > o)
> > min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
> > freemin=264669, free-target=352892, inactive-target=0, wired-max=2646699
> > faults=23269, traps=12279, intrs=37362874, ctxswitch=123098031 fpuswitch=0
> > softint=3819651, syscalls=122240657, kmapent=10
> > fault counts:
> > noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
> > ok relocks(total)=1288(1288), anget(retries)=17440(0), amapcopy=1892
> > neighbor anon/obj pg=3545/4472, gets(lock/unlock)=3091/1289
> > cases: anon=16354, anoncow=1086, obj=2586, prcopy=504, przero=2738
> > daemon and swap counts:
> > woke=0, revs=0, scans=0, obscans=0, anscans=0
> > busy=0, freed=0, reactivate=0, deactivate=0
> > pageouts=0, pending=0, nswget=0
> > nswapdev=1
> > swpages=18089189, swpginuse=0, swpgonly=0 paging=0
> > kernel pointers:
> > objs(kern)=0x8259ef18
> > ddb{0}> show bcstats
> > Current Buffer Cache status:
> > numbufs 777914 busymapped 0, delwri 0
> > kvaslots 6553 avail kva slots 6553
> > bufpages 3111642, dmapages 92181, dirtypages 0
> > pendingreads 0, pendingwrites 0
> > highflips 1259102, highflops 0, dmaflips 402
> > ddb{0}> machine ddbcpu 1
> > Stopped at x86_ipi_db+0x16:leave
> > x86_ipi_db(800048999ff0) at x86_ipi_db+0x16
> > x86_ipi_handler() at x86_ipi_handler+0x80
> > Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
> > acpicpu_idle() at acpicpu_idle+0x11f
> > sched_idle(800048999ff0) at sched_idle+0x282
> > end trace frame: 0x0, count: 10
> > ddb{1}> machine ddbcpu 2
> > Stopped at x86_ipi_db+0x16:leave
> > x86_ipi_db(8000489a2ff0) at x86_ipi_db+0x16
> > x86_ipi_handler() at x86_ipi_handler+0x80
> > Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
> > __mp_acquire_count(82613a98,1) at __mp_acquire_count+0x92
> > mi_switch() at mi_switch+0x301
> >
Re: panic - mutex? - AMD64/7.4 to current
On 24/08/24(Sat) 12:09, Hugh Graham wrote:
> The machine that slowly received the ports tree crashed upon reboot,
> and I have included the traces. It was running 7.5-stable at the
> time. I will wait an hour or so, and then restart and repeat
> against current.
Thanks, could you elaborate how you trigger the corruptions? I'd like
to try to reproduce them locally.
> # reboot
> dd: /var/db/host.random: No such file or directory
> chmod: /var/db/host.random: No such file or directory
> dd: /var/db/host.random: No such file or directory
> single user: not running shutdown scripts
> syncing disks...uvm_fault(0xfd880d4cb420, 0x40018, 0, 1) -> e
> kernel: page fault trap, code=0
> Stopped at _rb_remove+0x2e7: cmpl$0,0x18(%rax)
> TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
> *101771 59108 0 0x803 00K reboot
> 316626 12974 0 0x14000 0x42002 sensors
> _rb_remove(822f5390,fd8555e22ba8,fd854219dc48) at
> _rb_remove+0x
> 2e7
> brelse(fd854219dc48) at brelse+0x103
> vinvalbuf(fd8555e22b40,1,,80006a61c008,0,ff
> ff) at vinvalbuf+0x1e9
> vclean(fd8555e22b40,8,80006a61c008) at vclean+0xec
> vgonel(fd8555e22b40,80006a61c008) at vgonel+0x55
> vflush_vnode(fd8555e22b40,80006a64d730) at vflush_vnode+0x106
> vflush(801ccc00,0,2) at vflush+0x57
> ffs_flushfiles(801ccc00,2,80006a61c008) at ffs_flushfiles+0x8a
> ffs_unmount(801ccc00,8,80006a61c008) at ffs_unmount+0x3e
> dounmount_leaf(801ccc00,8,80006a61c008) at dounmount_leaf+0xbe
> dounmount(801ccc00,8,80006a61c008) at dounmount+0xec
> vfs_unmountall() at vfs_unmountall+0x7f
> vfs_shutdown(80006a61c008) at vfs_shutdown+0x3e
> boot(0) at boot+0x77
> end trace frame: 0x80006a64d9c0, count: 0
> https://www.openbsd.org/ddb.html describes the minimum info required in bug
> reports. Insufficient info makes it difficult to find and fix bugs.
> ddb{0}> show uvm
> Current UVM status:
> pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
> 7940097 VM pages: 98 active, 1215 inactive, 1 wired, 4713505 free (987509
> zer
> o)
> min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
> freemin=264669, free-target=352892, inactive-target=0, wired-max=2646699
> faults=23269, traps=12279, intrs=37362874, ctxswitch=123098031 fpuswitch=0
> softint=3819651, syscalls=122240657, kmapent=10
> fault counts:
> noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
> ok relocks(total)=1288(1288), anget(retries)=17440(0), amapcopy=1892
> neighbor anon/obj pg=3545/4472, gets(lock/unlock)=3091/1289
> cases: anon=16354, anoncow=1086, obj=2586, prcopy=504, przero=2738
> daemon and swap counts:
> woke=0, revs=0, scans=0, obscans=0, anscans=0
> busy=0, freed=0, reactivate=0, deactivate=0
> pageouts=0, pending=0, nswget=0
> nswapdev=1
> swpages=18089189, swpginuse=0, swpgonly=0 paging=0
> kernel pointers:
> objs(kern)=0x8259ef18
> ddb{0}> show bcstats
> Current Buffer Cache status:
> numbufs 777914 busymapped 0, delwri 0
> kvaslots 6553 avail kva slots 6553
> bufpages 3111642, dmapages 92181, dirtypages 0
> pendingreads 0, pendingwrites 0
> highflips 1259102, highflops 0, dmaflips 402
> ddb{0}> machine ddbcpu 1
> Stopped at x86_ipi_db+0x16:leave
> x86_ipi_db(800048999ff0) at x86_ipi_db+0x16
> x86_ipi_handler() at x86_ipi_handler+0x80
> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
> acpicpu_idle() at acpicpu_idle+0x11f
> sched_idle(800048999ff0) at sched_idle+0x282
> end trace frame: 0x0, count: 10
> ddb{1}> machine ddbcpu 2
> Stopped at x86_ipi_db+0x16:leave
> x86_ipi_db(8000489a2ff0) at x86_ipi_db+0x16
> x86_ipi_handler() at x86_ipi_handler+0x80
> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
> __mp_acquire_count(82613a98,1) at __mp_acquire_count+0x92
> mi_switch() at mi_switch+0x301
> sched_peg_curproc(8000489a2ff0) at sched_peg_curproc+0x65
> cpu_hz_update_sensor(8000489a2ff0) at cpu_hz_update_sensor+0x21
> sensor_task_work(8012ad00) at sensor_task_work+0x48
> taskq_thread(8018e080) at taskq_thread+0x100
> end trace frame: 0x0, count: 6
> ddb{2}> machine ddbcpu 3
> Stopped at x86_ipi_db+0x16:leave
> x86_ipi_db(8000489abff0) at x86_ipi_db+0x16
> x86_ipi_handler() at x86_ipi_handler+0x80
> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
> acpicpu_idle() at acpicpu_idle+0x11f
> sched_idle(8000489abff0) at sched_idle+0x282
> end trace frame: 0x0, count: 10
> ddb{3}> machine ddbcpu 4
> Stopped at x86_ipi_db+0x16:leave
> x86_ipi_db(8000489b4ff0) at x86_ipi_db+0x16
> x86_ipi_handler() at x86_ipi_handler+0x80
> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
> acpicpu_idle() at acpicpu_idle+0x11f
> sched_idle(8000489b4ff0) at sched_idle+0x282
> end trace frame: 0x0, count: 10
> ddb{4}> machine ddbcpu 5
> Stopped at
Re: panic - mutex? - AMD64/7.4 to current
The machine that slowly received the ports tree crashed upon reboot,
and I have included the traces. It was running 7.5-stable at the
time. I will wait an hour or so, and then restart and repeat
against current.
/Hugh
# reboot
dd: /var/db/host.random: No such file or directory
chmod: /var/db/host.random: No such file or directory
dd: /var/db/host.random: No such file or directory
single user: not running shutdown scripts
syncing disks...uvm_fault(0xfd880d4cb420, 0x40018, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at _rb_remove+0x2e7: cmpl$0,0x18(%rax)
TIDPIDUID PRFLAGS PFLAGS CPU COMMAND
*101771 59108 0 0x803 00K reboot
316626 12974 0 0x14000 0x42002 sensors
_rb_remove(822f5390,fd8555e22ba8,fd854219dc48) at _rb_remove+0x
2e7
brelse(fd854219dc48) at brelse+0x103
vinvalbuf(fd8555e22b40,1,,80006a61c008,0,ff
ff) at vinvalbuf+0x1e9
vclean(fd8555e22b40,8,80006a61c008) at vclean+0xec
vgonel(fd8555e22b40,80006a61c008) at vgonel+0x55
vflush_vnode(fd8555e22b40,80006a64d730) at vflush_vnode+0x106
vflush(801ccc00,0,2) at vflush+0x57
ffs_flushfiles(801ccc00,2,80006a61c008) at ffs_flushfiles+0x8a
ffs_unmount(801ccc00,8,80006a61c008) at ffs_unmount+0x3e
dounmount_leaf(801ccc00,8,80006a61c008) at dounmount_leaf+0xbe
dounmount(801ccc00,8,80006a61c008) at dounmount+0xec
vfs_unmountall() at vfs_unmountall+0x7f
vfs_shutdown(80006a61c008) at vfs_shutdown+0x3e
boot(0) at boot+0x77
end trace frame: 0x80006a64d9c0, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}> show uvm
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
7940097 VM pages: 98 active, 1215 inactive, 1 wired, 4713505 free (987509 zer
o)
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
freemin=264669, free-target=352892, inactive-target=0, wired-max=2646699
faults=23269, traps=12279, intrs=37362874, ctxswitch=123098031 fpuswitch=0
softint=3819651, syscalls=122240657, kmapent=10
fault counts:
noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
ok relocks(total)=1288(1288), anget(retries)=17440(0), amapcopy=1892
neighbor anon/obj pg=3545/4472, gets(lock/unlock)=3091/1289
cases: anon=16354, anoncow=1086, obj=2586, prcopy=504, przero=2738
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1
swpages=18089189, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0x8259ef18
ddb{0}> show bcstats
Current Buffer Cache status:
numbufs 777914 busymapped 0, delwri 0
kvaslots 6553 avail kva slots 6553
bufpages 3111642, dmapages 92181, dirtypages 0
pendingreads 0, pendingwrites 0
highflips 1259102, highflops 0, dmaflips 402
ddb{0}> machine ddbcpu 1
Stopped at x86_ipi_db+0x16:leave
x86_ipi_db(800048999ff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
acpicpu_idle() at acpicpu_idle+0x11f
sched_idle(800048999ff0) at sched_idle+0x282
end trace frame: 0x0, count: 10
ddb{1}> machine ddbcpu 2
Stopped at x86_ipi_db+0x16:leave
x86_ipi_db(8000489a2ff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
__mp_acquire_count(82613a98,1) at __mp_acquire_count+0x92
mi_switch() at mi_switch+0x301
sched_peg_curproc(8000489a2ff0) at sched_peg_curproc+0x65
cpu_hz_update_sensor(8000489a2ff0) at cpu_hz_update_sensor+0x21
sensor_task_work(8012ad00) at sensor_task_work+0x48
taskq_thread(8018e080) at taskq_thread+0x100
end trace frame: 0x0, count: 6
ddb{2}> machine ddbcpu 3
Stopped at x86_ipi_db+0x16:leave
x86_ipi_db(8000489abff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
acpicpu_idle() at acpicpu_idle+0x11f
sched_idle(8000489abff0) at sched_idle+0x282
end trace frame: 0x0, count: 10
ddb{3}> machine ddbcpu 4
Stopped at x86_ipi_db+0x16:leave
x86_ipi_db(8000489b4ff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
acpicpu_idle() at acpicpu_idle+0x11f
sched_idle(8000489b4ff0) at sched_idle+0x282
end trace frame: 0x0, count: 10
ddb{4}> machine ddbcpu 5
Stopped at x86_ipi_db+0x16:leave
x86_ipi_db(8000489bdff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
acpicpu_idle() at acpicpu_idle+0x11f
sched_idle(8000489bdff0) at sched_idle+0x282
end trace frame: 0x0, count: 10
ddb{5}> machine ddbcpu 6
Stopped at x86_ipi_db+0x16:
Re: panic - mutex? - AMD64/7.4 to current
On Sat, Aug 24, 2024 at 12:13:11PM +0200, Martin Pieuchot wrote: > Hugh, > > If you can reproduce this easily, please send a new panic with the > outputs of: > - show uvm > - show bcstats > - And the traces of all running processes... In the two reports below we > only have the trace of pax(1) which is running on CPU2. I saved screenshots of my first crash, with a few more traces in case they might be useful. These were running on 7.5-stable. http://www.oxide.org/amdcrash/ I ran my setup with a ports tree as the source overnight, and have not obtained the panic as yet. I will switch back to my cursed cpio and hopefully get the requested details on 7.6-beta today. When I try again to find a clean recipe, it will be multiple copies of CVSROOT. /Hugh > > The two panics are due to corruptions of two different global data > structures related to buffers: the tree of pages and the tree of buffers. > > In both cases it happens when the buffer cache reaches low DMA watermark > and tries to flip a buffer high. The fact that global data structures > are corrupted and the given buffer cannot be found tends to indicate > there is a race. And this is coherent with the use of pax | nc which > are currently running on two different CPUs. > > I fear there's a sleeping point somewhere, we could try converting the > splbio() to a mutex which should help. > > On 24/08/24(Sat) 01:11, Hugh Graham wrote: > > On Fri, Aug 23, 2024 at 01:52:52PM -0600, Bob Beck wrote: > > > My immediate suspicion would also fall there. Nothing in here has > > > recently changed. > > > > > > You should probably share this with a wider audience, like bugs@ or tech@ > > > instead of just > > > Mailing individuals. > > > > Apologies for the lack of process. I am only barely awake after a > > long slumber. > > > > It ran all day, but I did manage to reproduce the crash on 7.4, > > so that absolves a whole bunch of "recent" changes. > > > > Also, as yet, I have only the single machine for testing and > > can't exclude hardware. If anyone wants to make an independent > > confirmation, sending a ports tree with plenty of packages and > > distfiles might be a successful recipe. > > > > pax -w ports | network | pax -r > > > > Where the receiver's network media is forced to 10BaseT, or the > > sending machine is just that slow. My latest crash was near the > > 25GB mark, but this varies greatly and is usually sooner. I > > will confirm this recipe when I see my next crash. > > > > /Hugh > > > > >> OpenBSD/amd64 BOOTX64 3.65 > > boot> boot bsd.mp.74.dist -s > > booting hd0a:bsd.mp.74.dist: 17249612+4142096+368672+0+1241088 > > [1340407+128+1321080+1013316]=0x1973738 > > entry point at 0x1001000 > > [ using 3675960 bytes of bsd ELF symbol table ] > > Copyright (c) 1982, 1986, 1989, 1991, 1993 > > The Regents of the University of California. All rights reserved. > > Copyright (c) 1995-2023 OpenBSD. All rights reserved. > > https://www.OpenBSD.org > > > > OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023 > > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP > > real mem = 33551818752 (31997MB) > > ... > > panic: kernel diagnostic assertion "tpg != NULL" failed: file > > "/usr/src/sys/uvm > > /uvm_page.c", line 855 > > Stopped at db_enter+0x14: popq%rbp > > TIDPIDUID PRFLAGS PFLAGS CPU COMMAND > > * 1154 46918 00x13 02 pax > > 401298 12308 00x13 03 nc > > 404052 38135 0 0x14000 0x2001 softnet0 > > db_enter() at db_enter+0x14 > > panic(820a9e1f) at panic+0xc3 > > __assert(82122f8e,8207775c,357,8215caa2) at > > __assert+0x > > 29 > > uvm_pagerealloc_multi(fd8741fa1218,40,4000,22,8250d8d0) > > at u > > vm_pagerealloc_multi+0x2f8 > > buf_realloc_pages(fd8741fa1158,8250d8d0,2) at > > buf_realloc_pages+0xb > > f > > buf_flip_high(fd8741fa1158) at buf_flip_high+0x7e > > bufcache_recover_dmapages(0,4) at bufcache_recover_dmapages+0x12b > > buf_get(fd873280ab58,3be4,4000) at buf_get+0xcb > > getblk(fd873280ab58,3be4,4000,0,) at getblk+0x71 > > ffs2_balloc(fd872b47be18,ef9,2d,fd880dad9ea0,1,8000443a11a8) > > at > > ffs2_balloc+0xeef > > ffs_write(8000443a1228) at ffs_write+0x229 > > VOP_WRITE(fd873280ab58,8000443a1388,1,fd880dad9ea0) at > > VOP_WRITE+0x > > 45 > > vn_write(fd8718901708,8000443a1388,0) at vn_write+0xcc > > dofilewritev(80004436e2b0,6,8000443a1388,0,8000443a1460) at > > dofilew > > ritev+0x151 > > end trace frame: 0x8000443a13f0, count: 0 > > https://www.openbsd.org/ddb.html describes the minimum info required in bug > > reports. Insufficient info makes it difficult to find and fix bugs. > > ddb{2}> > > > > > > > > Very likely related to come of the changes being made in uvm. > > > > > > > On Aug 23,
