On 27/03/20(Fri) 22:43, Charlene Wendling wrote:
> Hi,
>
> >Environment:
> System : OpenBSD 6.6
> Details : OpenBSD 6.6-current (GENERIC.MP) #676: Fri Feb 14
> 02:26:37 MST 2020
> [email protected]:/usr/src/sys/arch/macppc/compile/GENERIC.MP
>
> Architecture: OpenBSD.macppc
> Machine : macppc
> >Description:
>
> Note that it's still reproducible with more recent snapshots.
>
> Running GENERIC.MP causes kernel panics if it's under high
> load. Running GENERIC causes no such issues on the two dual
> core machines belonging to the macppc ports building cluster.
>
> It's happening since early December 2019, but is occurring even
> more since the last few weeks, at a rate becoming harmful, hence my
> report.
>
> >How-To-Repeat:
>
> Start a bulk with dpb(1) with GENERIC.MP, it should panic anytime
> before 4 days. If you're lucky it will crash straight while listing
> ports.
Thanks for the report. If you have the patience to continue gather
such crash please do send the same report every time. It is
interesting to see that CPU0 is in uvm_swap_io() here.
It would be nice to know if there's a common pattern between what seems
to be a memory corruption on CPU1 and what CPU0 is doing at that moment.
This might be a MD or MI bug, so the more information you get us the
better :o)
>
> >Fix:
>
> None.
>
> --
>
> ddb{1}> machine ddbcpu 0
> Stopped at db_enter+0x10: lwz r0,36(r1)
> db_enter() at db_enter+0xc
> openpic_ipi_ddb() at openpic_ipi_ddb+0xc
> openpic_ext_intr() at openpic_ext_intr+0x254
> extint_call() at extint_call
> --- interrupt ---
> at 0xe000dffc
> ttyinput(e0005a00,e0008100) at ttyinput+0x8c
> zstty_rxsoft(6428,e0019000) at zstty_rxsoft+0x150
> zstty_softint(5ab65d38) at zstty_softint+0xb0
> zsc_intr_soft(e0000cd8) at zsc_intr_soft+0x7c
> zssoft(e0000cd8) at zssoft+0x64
> softintr_dispatch(e0000c00) at softintr_dispatch+0x80
> dosoftint(1) at dosoftint+0xa4
> openpic_splx(100) at openpic_splx+0xa4
> splx(65727000) at splx+0x1c
> end trace frame: 0xe629c780, count: 0
>
> ddb{0}> trace
> db_enter() at db_enter+0xc
> openpic_ipi_ddb() at openpic_ipi_ddb+0xc
> openpic_ext_intr() at openpic_ext_intr+0x254
> extint_call() at extint_call
> --- interrupt ---
> at 0xe000dffc
> ttyinput(e0005a00,e0008100) at ttyinput+0x8c
> zstty_rxsoft(6428,e0019000) at zstty_rxsoft+0x150
> zstty_softint(5ab65d38) at zstty_softint+0xb0
> zsc_intr_soft(e0000cd8) at zsc_intr_soft+0x7c
> zssoft(e0000cd8) at zssoft+0x64
> softintr_dispatch(e0000c00) at softintr_dispatch+0x80
> dosoftint(1) at dosoftint+0xa4
> openpic_splx(100) at openpic_splx+0xa4
> splx(65727000) at splx+0x1c
> tsleep(6428,920000,e629c7d0,0) at tsleep+0x98
> biowait(1) at biowait+0x5c
> uvm_swap_io(ffffffff,0,0,20000000) at uvm_swap_io+0x5f4
> uvm_swap_get(3e60590,3e60590,e629c8e0) at uvm_swap_get+0x58
> uvmfault_anonget(400,5,e629c930) at uvmfault_anonget+0x1ac
> uvm_fault(6ab1e668,40f8050,e629c970,20009034) at uvm_fault+0x554
> trap(6f3b63c8) at trap+0x68c
> trapagain() at trapagain+0x4
> --- trap (type 0x300) ---
> at 0xe629cbf0
> ureadc(e0005a00,0) at ureadc+0x128
> ttread(6ab49338,300,e629cc90) at ttread+0x368
> zsread(f4f958,40004048,1a2454c0) at zsread+0x58
> spec_read(fe2f60) at spec_read+0x354
> ufsspec_read(2001) at ufsspec_read+0x20
> VOP_READ(925e6c,f4f680,e629cdd0,0) at VOP_READ+0x50
> vn_read(1,1,e629ce20) at vn_read+0xc4
> dofilereadv(6ab49338,e629ce48,e629cec0,6ab49374,2e) at dofilereadv+0xd0
> sys_read(d891b0a8,6ab49374,e629cea4) at sys_read+0x64
> trap(6ab49338) at trap+0x9f0
> trapagain() at trapagain+0x4
> --- syscall (number 3) ---
> End of kernel: 0xfffcef70
> end trace frame: 0xfffcef70, count: -34
>
> ddb{0}> machine ddbcpu 1
> Stopped at db_enter+0x10: lwz r0,36(r1)
> db_enter() at db_enter+0xc
> panic(0) at panic+0xe0
> rw_assert_rdlock(e61f9e88) at rw_assert_rdlock+0x60
> rw_exit_read(9737f8) at rw_exit_read+0x1c
> if_input_process(792280,e61f9f28) at if_input_process+0x68
> ifiq_process(ffffffff) at ifiq_process+0x78
> taskq_thread(e0007040) at taskq_thread+0x58
> fork_trampoline() at fork_trampoline+0x14
> end trace frame: 0x0, count: 7
>
> ddb{1}> trace
> db_enter() at db_enter+0xc
> panic(0) at panic+0xe0
> rw_assert_rdlock(e61f9e88) at rw_assert_rdlock+0x60
> rw_exit_read(9737f8) at rw_exit_read+0x1c
> if_input_process(792280,e61f9f28) at if_input_process+0x68
> ifiq_process(ffffffff) at ifiq_process+0x78
> taskq_thread(e0007040) at taskq_thread+0x58
> fork_trampoline() at fork_trampoline+0x14
> end trace frame: 0x0, count: -8
>
> ddb{0}> show uvm
> Current UVM status:
> pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
> 505886 VM pages: 24678 active, 41042 inactive, 0 wired, 321164 free
> (40160 ze
> ro) min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
> freemin=16862, free-target=22482, inactive-target=117030,
> wired-max=168628 faults=1624317059, traps=0, intrs=65203768,
> ctxswitch=168296430 fpuswitch=179
> 43977 softint=79503845, syscalls=1341239677, kmapent=10
> fault counts:
> noram=0, noanon=0, noamap=0, pgwait=5, pgrele=0
> ok relocks(total)=4186032(4186083),
> anget(retries)=445957253(2259963), amap
> copy=474564238 neighbor anon/obj pg=56139479/858277021,
> gets(lock/unlock)=251501358/192612
> 1 cases: anon=384161854, anoncow=61795398, obj=206418754,
> prcopy=45082552, pr
> zero=926858494 daemon and swap counts:
> woke=688, revs=687, scans=33611484, obscans=221741, anscans=33389743
> busy=0, freed=3420587, reactivate=0, deactivate=10674761
> pageouts=1887263, pending=145871, nswget=2260009
> nswapdev=1
> swpages=589823, swpginuse=3448, swpgonly=2520 paging=0
> kernel pointers:
> objs(kern)=0x964b78
>
> ddb{0}> show bcstats
> Current Buffer Cache status:
> numbufs 39183 busymapped 2, delwri 481
> kvaslots 358 avail kva slots 356
> bufpages 101112, dmapages 101112, dirtypages 1924
> pendingreads 56, pendingwrites 487
> highflips 0, highflops 0, dmaflips 0
>
> ddb{0}> ps
> PID TID PPID UID S FLAGS WAIT COMMAND
> 13838 77554 53101 55 2 0x100002 tar
> 5 411345 53101 55 3 0x100002 getblk gzip
> 53101 519994 76451 55 3 0x10008a pause sh
> 76451 100162 4617 55 3 0x10008a pause make
> 72529 347382 47236 55 3 0x100082 piperd as
> 60196 163037 47236 55 2 0x2 cc1plus
> 47236 323488 65195 55 3 0x82 wait eg++
> 65195 220436 68022 55 3 0x10008a pause sh
> 4617 88244 90890 55 3 0x10008a pause make
> 90890 445970 80586 55 3 0x10008a pause sh
> 80586 326053 9739 55 3 0x10008a pause make
> 9739 263861 82187 55 3 0x10008a pause sh
> 82187 321585 7869 55 3 0x10008a pause make
> 7869 105973 49600 0 3 0x10008a pause ksh
> 68022 355676 34277 55 3 0x10008a pause make
> 34277 368601 88892 55 3 0x10008a pause sh
> 88892 305297 56803 55 3 0x200082 piperd ruby26
> 88892 393793 56803 55 3 0x4200082 poll ruby26
> 56803 227831 2148 55 3 0x10008a pause sh
> 2148 58486 81361 55 3 0x10008a pause make
> 81361 338563 77685 55 3 0x10008a pause make
> 77685 388786 63413 55 3 0x10008a pause sh
> 63413 71466 27598 55 3 0x10008a pause make
> 27598 228483 84189 55 3 0x10008a pause sh
> 84189 327534 85052 55 3 0x10008a pause make
> 85052 331567 49600 0 3 0x10008a pause ksh
> 99892 361800 92768 0 3 0x92 select sshd
> 49600 61156 92768 0 3 0x92 select sshd
> *45234 342510 1 0 7 0x100003 getty
> 93075 451162 1 0 3 0x100098 poll cron
> 40632 294334 68073 95 3 0x100092 kqread smtpd
> 66378 222405 68073 103 3 0x100092 kqread smtpd
> 29268 40590 68073 95 3 0x100092 kqread smtpd
> 83712 109844 68073 95 3 0x100092 kqread smtpd
> 73028 492777 68073 95 3 0x100092 kqread smtpd
> 88466 77882 68073 95 3 0x100092 kqread smtpd
> 68073 465851 1 0 3 0x100080 kqread smtpd
> 92768 193425 1 0 3 0x80 select sshd
> 70577 170085 0 0 3 0x14280 nfsidl nfsio
> 72396 173828 0 0 3 0x14200 netio nfsio
> 90067 72030 0 0 3 0x14280 nfsidl nfsio
> 30486 191183 0 0 2 0x14280 nfsio
> 47309 225822 1 28 3 0x100090 poll portmap
> 80188 54001 82663 83 3 0x100092 poll ntpd
> 82663 79262 79579 83 3 0x100092 poll ntpd
> 79579 235914 1 0 3 0x100080 poll ntpd
> 46374 201019 4846 74 3 0x100092 bpf pflogd
> 4846 152817 1 0 3 0x80 netio pflogd
> 90443 100 97507 73 3 0x100090 kqread syslogd
> 97507 229136 1 0 3 0x100082 netio syslogd
> 52873 191148 54126 115 3 0x100092 kqread slaacd
> 30818 333418 54126 115 3 0x100092 kqread slaacd
> 54126 442565 1 0 3 0x100080 kqread slaacd
> 88287 27153 0 0 3 0x40014200 idle1
> 27090 22714 0 0 2 0x14200 zerothread
> 27203 326082 0 0 3 0x14200 aiodoned aiodoned
> 59133 516141 0 0 3 0x14200 syncer update
> 39764 425180 0 0 3 0x14200 cleaner cleaner
> 31597 511105 0 0 3 0x14200 reaper reaper
> 92937 251850 0 0 3 0x14200 pgdaemon pagedaemon
> 93164 179593 0 0 3 0x14200 bored crynlk
> 69559 381744 0 0 3 0x14200 bored crypto
> 63543 457414 0 0 3 0x14200 usbtsk usbtask
> 84397 184135 0 0 3 0x14200 usbatsk usbatsk
> 94651 47216 0 0 3 0x14200 blinken xlights0
> 17948 337515 0 0 3 0x14200 bored sensors
> 79162 178788 0 0 7 0x14200 softnet
> 16453 109484 0 0 3 0x14200 bored systqmp
> 83069 51784 0 0 3 0x14200 bored systq
> 86874 289725 0 0 2 0x40014200 softclock
> 73075 431015 0 0 3 0x40014200 idle0
> 61590 3767 0 0 3 0x14200 bored smr
> 1 469012 0 0 3 0x82 wait init
> 0 0 -1 0 3 0x10200 scheduler swapper
> --
> dmesg:
>
> OpenBSD 6.6-current (GENERIC.MP) #676: Fri Feb 14 02:26:37 MST 2020
> [email protected]:/usr/src/sys/arch/macppc/compile/GENERIC.MP
> real mem = 2147483648 (2048MB)
> avail mem = 2072006656 (1976MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root: model RackMac1,1
> cpu0 at mainbus0: 7455 (Revision 0x201): 999 MHz: 256KB L2 cache, 2MB
> L3 cache cpu1 at mainbus0: 7455 (Revision 0x201): 999 MHz: 256KB L2
> cache, 2MB L3 cache mem0 at mainbus0
> spdmem0 at mem0: 512MB DDR SDRAM non-parity PC2100CL2.5
> spdmem1 at mem0: 512MB DDR SDRAM non-parity PC2100CL2.5
> spdmem2 at mem0: 512MB DDR SDRAM non-parity PC2100CL2.5
> spdmem3 at mem0: 512MB DDR SDRAM non-parity PC2100CL2.5
> memc0 at mainbus0: uni-n rev 0x24
> kiic0 at memc0 offset 0xf8001000
> iic0 at kiic0
> lmenv0 at iic0 addr 0xad: lm87 rev 6
> lmtemp0 at iic0 addr 0x49: ds1775
> mpcpcibr0 at mainbus0 pci: uni-north
> pci0 at mpcpcibr0 bus 0
> mpcpcibr1 at mainbus0 pci: uni-north
> pci1 at mpcpcibr1 bus 0
> ppb0 at pci1 dev 13 function 0 "Intel 21154AE/BE" rev 0x00
> pci2 at ppb0 bus 1
> macobio0 at pci2 dev 7 function 0 "Apple Keylargo" rev 0x03
> openpic0 at macobio0 offset 0x40000: version 0x4614 feature 3f0302 LE
> macgpio0 at macobio0 offset 0x50
> macgpio1 at macgpio0 offset 0x9: irq 47
> "programmer-switch" at macgpio0 offset 0x11 not configured
> "ringDetect-gpio" at macgpio0 offset 0x8 not configured
> "keySwitch-gpio" at macgpio0 offset 0xc not configured
> "systemMonitor-gpio" at macgpio0 offset 0x12 not configured
> sysbutton0 at macgpio0 offset 0x15: irq 59
> "indicatorLED-gpio" at macgpio0 offset 0x20 not configured
> "virtual-sound" at macgpio0 not configured
> "escc-legacy" at macobio0 offset 0x12000 not configured
> zs0 at macobio0 offset 0x13000: irq 22,23
> zstty0 at zs0 channel 0: console
> zstty1 at zs0 channel 1
> xlights0 at macobio0 offset 0x10000: irq 1
> "timer" at macobio0 offset 0x15000 not configured
> adb0 at macobio0 offset 0x16000
> apm0 at adb0: battery flags 0x9, 0% charged
> piic0 at adb0
> iic1 at piic0
> "PCA9554" at iic1 addr 0xa0 not configured
> "PCA9554" at iic1 addr 0xa1 not configured
> "PCA9554" at iic1 addr 0xa2 not configured
> "PCA9554" at iic1 addr 0xa3 not configured
> "PCA9554" at iic1 addr 0xa4 not configured
> kiic1 at macobio0 offset 0x18000
> iic2 at kiic1
> wdc0 at macobio0 offset 0x1f000 irq 19: DMA
> atapiscsi0 at wdc0 channel 0 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <LG, CD-ROM CRN-8245B, AHT9> removable
> cd0(wdc0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 2
> ohci0 at pci2 dev 8 function 0 "Apple USB" rev 0x00: irq 27, version 1.0
> ohci1 at pci2 dev 9 function 0 "Apple USB" rev 0x00: irq 28, version 1.0
> usb0 at ohci0: USB revision 1.0
> uhub0 at usb0 configuration 1 interface 0 "Apple OHCI root hub" rev
> 1.00/1.00 addr 1 usb1 at ohci1: USB revision 1.0
> uhub1 at usb1 configuration 1 interface 0 "Apple OHCI root hub" rev
> 1.00/1.00 addr 1 ppb1 at pci1 dev 17 function 0 "Intel 21154AE/BE" rev
> 0x00 pci3 at ppb1 bus 2
> pciide0 at pci1 dev 21 function 0 "Promise PDC20268R" rev 0x02: DMA,
> channel 0 configured to native-PCI, channel 1 configured to native-PCI
> pciide0: using irq 58 for native-PCI interrupt wd0 at pciide0 channel 0
> drive 0: <WL500GPA1672> wd0: 16-sector PIO, LBA48, 476940MB, 976773168
> sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
> wd1 at pciide0 channel 1 drive 1: <ST3160215ACE>
> wd1: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
> wd1(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 5
> pciide1 at pci1 dev 27 function 0 "Promise PDC20268R" rev 0x02: DMA,
> channel 0 configured to native-PCI, channel 1 configured to native-PCI
> pciide1: using irq 63 for native-PCI interrupt mpcpcibr2 at mainbus0
> pci: uni-north pci4 at mpcpcibr2 bus 0
> "Apple UniNorth Firewire" rev 0x01 at pci4 dev 14 function 0 not
> configured gem0 at pci4 dev 15 function 0 "Apple Uni-N2 GMAC" rev 0x00:
> irq 41, address 00:03:5f:ab:07:4a brgphy0 at gem0 phy 0: BCM5421
> 10/100/1000baseT PHY, rev. 1 vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> bootpath: /pci@f2000000/AppleKiwi@15/ata-6@0/disk@0:/bsd
> root on wd0a (e568d7921a119aae.a) swap on wd0b dump on wd0b
> WARNING: / was not properly unmounted
>
