Hi,
My armv7 regress machine hangs every day in regress/sys/kern/fork-exit.
I can still break into ddb. Looks like an endless loop in
uvm_map_fix_space().
Note that it does not happen when I boot the machine and run only
the fork-exit test. This test hangs it when I run all regress tests
in a row. I found a "thread: table is full" in dmesg, maybe it is
related.
This problem exists since I implemented the fork-exit test. But
it works on all my other architectures. (i386 amd64 arm64 powerpc64
sparc64)
bluhm
Stopped at db_enter: ldrb r15, [r15, r15, ror r15]!
ddb> trace
db_enter
rlv=0xc03f3a70 rfp=0xcea1fe08
ampintc_irq_handler+0x13c
rlv=0xc050a840 rfp=0xcea1fe70
irq_entry+0x78
rlv=0xc04cb178 rfp=0xcea1fe98
uvm_map_fix_space+0x204
rlv=0xc04caabc rfp=0xcea1fec0
uvm_map_kmem_grow+0x154
rlv=0xc04ca2d0 rfp=0xcea1ff48
uvm_map+0x32c
rlv=0xc063a0ac rfp=0xcea1ffa8
uvm_km_thread+0x10c
rlv=0xc0655a8c rfp=0xc0a4ff50
Bad frame pointer: 0xc0a4ff50
ddb> c
Stopped at db_enter: ldrb r15, [r15, r15, ror r15]!
ddb> trace
db_enter
rlv=0xc03f3a70 rfp=0xcea1fe08
ampintc_irq_handler+0x13c
rlv=0xc050a840 rfp=0xcea1fe70
irq_entry+0x78
rlv=0xc04cb178 rfp=0xcea1fe98
uvm_map_fix_space+0x204
rlv=0xc04caabc rfp=0xcea1fec0
uvm_map_kmem_grow+0x154
rlv=0xc04ca2d0 rfp=0xcea1ff48
uvm_map+0x32c
rlv=0xc063a0ac rfp=0xcea1ffa8
uvm_km_thread+0x10c
rlv=0xc0655a8c rfp=0xc0a4ff50
Bad frame pointer: 0xc0a4ff50
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
31447 184367 47592 0 2 0x480 fork-exit
84316 242076 47592 0 2 0 fork-exit
42883 265234 47592 0 2 0 fork-exit
9618 65399 47592 0 2 0x480 fork-exit
19737 217458 47592 0 2 0 fork-exit
57604 238433 47592 0 2 0 fork-exit
83472 404465 47592 0 2 0 fork-exit
35282 370489 47592 0 2 0x480 fork-exit
23531 101851 47592 0 2 0 fork-exit
18930 88298 47592 0 2 0 fork-exit
68999 305925 47592 0 2 0x480 fork-exit
13860 474368 47592 0 2 0 fork-exit
33789 500409 47592 0 2 0 fork-exit
94851 254198 47592 0 2 0x480 fork-exit
57014 194240 47592 0 2 0 fork-exit
75921 38587 47592 0 2 0 fork-exit
97688 494838 47592 0 2 0x480 fork-exit
91843 28636 47592 0 2 0x480 fork-exit
97284 19654 47592 0 2 0 fork-exit
74428 210286 47592 0 2 0 fork-exit
76767 378923 47592 0 2 0x480 fork-exit
78209 65752 47592 0 2 0 fork-exit
16736 517887 47592 0 2 0x480 fork-exit
95889 454303 47592 0 2 0 fork-exit
80557 70159 47592 0 2 0 fork-exit
11542 165922 47592 0 2 0x480 fork-exit
66553 382503 47592 0 2 0 fork-exit
50470 467992 47592 0 2 0 fork-exit
1402 347271 47592 0 2 0 fork-exit
44781 253504 47592 0 2 0x480 fork-exit
50049 375783 47592 0 2 0x480 fork-exit
64808 330811 47592 0 2 0 fork-exit
61698 33676 47592 0 2 0 fork-exit
99140 12062 47592 0 2 0 fork-exit
72835 320538 47592 0 2 0 fork-exit
97276 426323 47592 0 2 0 fork-exit
23513 435315 47592 0 2 0 fork-exit
19769 460679 47592 0 2 0x480 fork-exit
81088 521689 47592 0 2 0x480 fork-exit
18492 34938 47592 0 2 0x480 fork-exit
34061 321358 47592 0 2 0 fork-exit
56215 360545 47592 0 2 0x480 fork-exit
83814 205792 47592 0 2 0x480 fork-exit
60183 346225 47592 0 2 0x480 fork-exit
75018 192517 47592 0 2 0x480 fork-exit
77465 257309 47592 0 2 0x480 fork-exit
85628 49351 47592 0 2 0x480 fork-exit
49674 436828 47592 0 2 0 fork-exit
39999 259378 47592 0 2 0x480 fork-exit
9256 255847 47592 0 2 0 fork-exit
50581 350549 47592 0 2 0 fork-exit
35476 256335 47592 0 2 0x480 fork-exit
39278 358565 47592 0 2 0x480 fork-exit
29102 409232 47592 0 2 0x480 fork-exit
24869 50864 47592 0 2 0x480 fork-exit
60131 33285 47592 0 2 0x480 fork-exit
73184 310151 47592 0 2 0 fork-exit
94124 496847 47592 0 2 0x480 fork-exit
35275 417281 47592 0 2 0x480 fork-exit
85113 4276 47592 0 2 0x480 fork-exit
27889 145051 47592 0 2 0x480 fork-exit
48864 346613 47592 0 2 0x480 fork-exit
21922 333727 47592 0 2 0 fork-exit
41059 523807 47592 0 2 0x480 fork-exit
93743 336096 47592 0 2 0x480 fork-exit
98030 157973 47592 0 2 0x480 fork-exit
74662 316962 47592 0 2 0x480 fork-exit
51094 293522 47592 0 2 0x480 fork-exit
20985 427077 47592 0 2 0 fork-exit
56526 15934 47592 0 2 0x480 fork-exit
10491 460342 47592 0 2 0 fork-exit
1795 35243 47592 0 2 0x480 fork-exit
86985 326404 47592 0 2 0x480 fork-exit
3930 490803 47592 0 2 0x480 fork-exit
9392 22539 47592 0 2 0 fork-exit
48609 430465 47592 0 2 0 fork-exit
37105 312869 47592 0 2 0x480 fork-exit
31715 129967 47592 0 2 0x480 fork-exit
23921 15643 47592 0 2 0x480 fork-exit
77893 322524 47592 0 2 0x480 fork-exit
11041 24714 47592 0 2 0x480 fork-exit
403 466824 47592 0 2 0x480 fork-exit
88409 56831 47592 0 2 0x480 fork-exit
30370 216739 47592 0 2 0x480 fork-exit
81200 375221 47592 0 2 0x480 fork-exit
50774 100673 47592 0 2 0x480 fork-exit
34544 278865 47592 0 2 0x480 fork-exit
76004 407439 47592 0 2 0x480 fork-exit
60888 428150 47592 0 2 0x480 fork-exit
81689 198650 47592 0 2 0x480 fork-exit
54314 221791 47592 0 2 0x480 fork-exit
59550 1036 47592 0 2 0x480 fork-exit
68241 193946 47592 0 2 0x480 fork-exit
49349 119833 47592 0 2 0x480 fork-exit
38650 173366 47592 0 2 0x480 fork-exit
46691 96355 47592 0 2 0x480 fork-exit
76659 210094 47592 0 2 0x480 fork-exit
57396 489703 47592 0 2 0x480 fork-exit
29893 158801 47592 0 2 0x480 fork-exit
11175 492268 47592 0 2 0x480 fork-exit
47592 124718 64559 0 2 0x482 fork-exit
64559 386272 20394 0 3 0x10008a sigsusp sh
20394 24816 31906 0 3 0x10008a sigsusp make
31906 14291 14162 0 3 0x10008a sigsusp sh
14162 306946 97992 0 3 0x10008a sigsusp make
98858 331997 0 0 2 0x14200 sensors
67212 51498 24093 0 3 0x100082 piperd gzip
24093 265646 97992 0 3 0x100082 piperd pax
97992 159549 36907 0 2 0x82 perl
36907 169313 57055 0 3 0x10008a sigsusp ksh
57055 415743 98515 0 3 0x9a select sshd
73200 412748 1 0 2 0x100083 getty
26099 300396 1 0 2 0x100498 cron
95985 298231 1 99 3 0x100090 poll sndiod
70278 446998 1 110 3 0x100090 poll sndiod
55031 184969 83699 95 3 0x100092 kqread smtpd
77424 166550 83699 103 3 0x100092 kqread smtpd
89398 274829 83699 95 3 0x100092 kqread smtpd
59405 342151 83699 95 3 0x100092 kqread smtpd
13268 136277 83699 95 3 0x100092 kqread smtpd
38027 546 83699 95 3 0x100092 kqread smtpd
83699 393182 1 0 3 0x100080 kqread smtpd
98515 274114 1 0 3 0x88 select sshd
88595 204229 0 0 2 0x14600 acct
45496 159948 0 0 3 0x14280 nfsidl nfsio
58596 516623 0 0 3 0x14280 nfsidl nfsio
24733 477650 0 0 3 0x14280 nfsidl nfsio
598 87508 0 0 3 0x14280 nfsidl nfsio
49126 292186 1 0 3 0x100080 poll ntpd
19035 233928 48722 83 3 0x100092 poll ntpd
48722 270765 1 83 2 0x100492 ntpd
10218 415954 43067 74 2 0x100492 pflogd
43067 135020 1 0 3 0x80 netio pflogd
432 326804 21955 73 2 0x100490 syslogd
21955 57477 1 0 3 0x100082 netio syslogd
22335 405879 1 0 3 0x100080 kqread resolvd
75354 226031 5825 77 3 0x100092 kqread dhcpleased
84988 337810 5825 77 3 0x100092 kqread dhcpleased
5825 42668 1 0 3 0x80 kqread dhcpleased
27305 391368 70568 115 3 0x100092 kqread slaacd
98634 200814 70568 115 3 0x100092 kqread slaacd
70568 269180 1 0 3 0x100080 kqread slaacd
95757 10364 0 0 3 0x14200 bored smr
93105 135251 0 0 2 0x14200 zerothread
64224 325787 0 0 3 0x14200 aiodoned aiodoned
34531 22177 0 0 2 0x14600 update
78687 265567 0 0 3 0x14200 cleaner cleaner
232 11904 0 0 3 0x14200 reaper reaper
74714 96620 0 0 3 0x14200 pgdaemon pagedaemon
64939 395318 0 0 3 0x14200 bored bwfm0
35236 26500 0 0 3 0x14200 bored crynlk
61298 434667 0 0 3 0x14200 bored crypto
91567 70785 0 0 3 0x14200 mmctsk sdmmc1
5671 375934 0 0 3 0x14200 mmctsk sdmmc0
83597 416341 0 0 3 0x14200 usbtsk usbtask
35993 32256 0 0 3 0x14200 usbatsk usbatsk
4826 202918 0 0 2 0x14200 softnet
29716 186612 0 0 2 0x14200 systqmp
10929 285830 0 0 2 0x14200 systq
60621 149663 0 0 2 0x40014200 softclock
30603 410944 0 0 3 0x40014200 idle0
*95699 307018 0 0 7 0x14200 kmthread
1 471814 0 0 3 0x82 wait init
0 0 -1 0 2 0x10200 swapper
ddb> show register
spsr 0x20000113
r0 0x1
r1 0x40 fiqvector+0x24
r2 0
r3 0xc0651b3c armv7_bs_r_2
r4 0xc85d5800 end+0x7c9828c
r5 0xc86bc600 end+0x7d7f08c
r6 0xc85d5848 end+0x7c982d4
r7 0xc85d58d4 end+0x7c98360
r8 0xc85d544c end+0x7c97ed8
r9 0xce9af000 end+0xe071a8c
r10 0xc092c648 db_console
r11 0xcea1fde0 end+0xe0e286c
r12 0
usr_sp 0xbffcddf0
usr_lr 0x5dd60924
svc_sp 0xcea1fdc0 end+0xe0e284c
svc_lr 0xc05394c0 imxuart_intr+0x1b0
pc 0xc06ebde4 db_enter
und_sp 0xcea1eff0 end+0xe0e1a7c
abt_sp 0xc0a4d000 end+0x10fa8c
irq_sp 0xc0a4c000 end+0x10ea8c
db_enter: ldrb r15, [r15, r15, ror r15]!
ddb> dmesg
OpenBSD 7.0-current (GENERIC) #80412: Mon Oct 11 23:54:43 MDT 2021
[email protected]:/usr/src/sys/arch/armv7/compile/GENERIC
real mem = 2101563392 (2004MB)
avail mem = 2050285568 (1955MB)
random: good seed from bootblocks
mainbus0 at root: SolidRun Cubox-i Dual/Quad
cpu0 at mainbus0 mpidr 0: ARM Cortex-A9 r2p10
cpu0: 32KB 32b/line 4-way L1 VIPT I-cache, 32KB 32b/line 4-way L1 D-cache
cortex0 at mainbus0
amptimer0 at cortex0: 396000 kHz
armliicc0 at cortex0: rtl 7 waymask: 0x0000000f
simplebus0 at mainbus0: "soc"
ampintc0 at simplebus0 nirq 160, ncpu 4: "interrupt-controller"
"dma-apbh" at simplebus0 not configured
"hdmi" at simplebus0 not configured
"gpu" at simplebus0 not configured
"gpu" at simplebus0 not configured
"timer" at simplebus0 not configured
"cache-controller" at simplebus0 not configured
simplebus1 at simplebus0: "bus"
imxccm0 at simplebus1
imxanatop0 at simplebus1
syscon0 at simplebus1: "snvs"
imxrtc0 at syscon0
"snvs-lpgpr" at syscon0 not configured
imxsrc0 at simplebus1
syscon1 at simplebus1: "iomuxc-gpr"
"mux-controller" at syscon1 not configured
"ipu1_csi0_mux" at syscon1 not configured
"ipu2_csi1_mux" at syscon1 not configured
imxiomuxc0 at simplebus1
simplebus2 at simplebus1: "spba-bus"
"spdif" at simplebus2 not configured
imxuart0 at simplebus2: console
"asrc" at simplebus2 not configured
"vpu" at simplebus1 not configured
"pwm" at simplebus1 not configured
"timer" at simplebus1 not configured
imxgpio0 at simplebus1
imxgpio1 at simplebus1
imxgpio2 at simplebus1
imxgpio3 at simplebus1
imxgpio4 at simplebus1
imxgpio5 at simplebus1
imxgpio6 at simplebus1
imxdog0 at simplebus1
"usbphy" at simplebus1 not configured
"usbphy" at simplebus1 not configured
imxgpc0 at simplebus1
"sdma" at simplebus1 not configured
simplebus3 at simplebus0: "bus"
syscon2 at simplebus3: "efuse"
"crypto" at simplebus3 not configured
imxehci0 at simplebus3
usb0 at imxehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "i.MX EHCI root hub" rev 2.00/1.00
addr 1
imxehci1 at simplebus3
usb1 at imxehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "i.MX EHCI root hub" rev 2.00/1.00
addr 1
"usbmisc" at simplebus3 not configured
fec0 at simplebus3
fec0: address d0:63:b4:00:98:5f
atphy0 at fec0 phy 0: AR8035 10/100/1000 PHY, rev. 2
imxesdhc0 at simplebus3
imxesdhc0: 198 MHz base clock
sdmmc0 at imxesdhc0: 4-bit, sd high-speed, mmc high-speed, dma
imxesdhc1 at simplebus3
imxesdhc1: 198 MHz base clock
sdmmc1 at imxesdhc1: 4-bit, sd high-speed, mmc high-speed, dma
imxiic0 at simplebus3
iic0 at imxiic0
imxiic1 at simplebus3
iic1 at imxiic1
pcfrtc0 at iic1 addr 0x68: battery ok
"memory-controller" at simplebus3 not configured
"vdoa" at simplebus3 not configured
imxuart1 at simplebus3
"ipu" at simplebus0 not configured
"sram" at simplebus0 not configured
imxahci0 at simplebus0: AHCI 1.3
imxahci0: port 0: 3.0Gb/s
scsibus0 at imxahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, Crucial_CT128M55, MU01> naa.500a07510c8d1a6f
sd0: 122104MB, 512 bytes/sector, 250069680 sectors, thin
"gpu" at simplebus0 not configured
"ipu" at simplebus0 not configured
scsibus1 at sdmmc1: 2 targets, initiator 0
sd1 at scsibus1 targ 1 lun 0: <SD/MMC, SL08G, 0080> removable
sd1: 7580MB, 512 bytes/sector, 15523840 sectors
bwfm0 at sdmmc0 function 1
manufacturer 0x02d0, product 0x4330 at sdmmc0 function 2 not configured
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
bootfile: sd0a:/bsd
boot device: sd0
root on sd0a (6e54f9212dcb6429.a) swap on sd0b dump on sd0b
bwfm0: address 6c:ad:f8:1d:45:4f
[10-1.test]71586/149217 sp=7ffdbe08 inside dead4110-dead4110: not MAP_STACK
[3-2.test]14876/20270 sp=86581e90 inside 86582000-86583000: not MAP_STACK
<3>process: table is full
free inode /usr/obj/596822 had 2112 blocks
free inode /usr/obj/596834 had 2144 blocks
free inode /usr/obj/597090 had 2144 blocks
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 0, 006>
sd2: 3299MB, 512 bytes/sector, 6756480 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 0, 006>
sd2: 3299MB, 512 bytes/sector, 6756480 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 0, 006>
sd2: 3299MB, 512 bytes/sector, 6756480 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 5, 006>
sd2: 2199MB, 512 bytes/sector, 4504320 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 5, 006>
sd2: 2199MB, 512 bytes/sector, 4504320 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 5, 006>
sd2: 2199MB, 512 bytes/sector, 4504320 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CONCAT, 006>
sd2: 3299MB, 512 bytes/sector, 6756816 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CONCAT, 006>
sd2: 3299MB, 512 bytes/sector, 6756816 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CONCAT, 006>
sd2: 3299MB, 512 bytes/sector, 6756816 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1C, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1C, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1C, 006>
sd2: 1099MB, 512 bytes/sector, 2252272 sectors
sd2 detached
<3>uid 0 on /mnt/regress-ffs/fstest_bf9f239dd5c566e7b5afa3a3a45af994: file
system full
<3>uid 0 on /mnt/regress-ffs/fstest_3d885fb755de2bdacd4cc6abf104949f: out of
inodes
<3>uid 0 on /mnt/regress-ffs/fstest_db440b8b2fd17687db6800dffe258308: out of
inodes
<3>uid 0 on /mnt/regress-ffs/fstest_f2f321d3ede3fd9b84302d0e1bd7bce9: out of
inodes
<3>uid 0 on /mnt/regress-ffs/fstest_2ed0e88ddbc8fe4578dab4f9edc819c8: out of
inodes
<3>uid 0 on /mnt/regress-nfs-client/fstest_09c54caf181311d3877ef0a6f39f6c97:
file system full
<3>uid 0 on /mnt/regress-nfs-client/fstest_6377cecb851ecea2a1cdd4f68fd8b293:
out of inodes
<3>uid 0 on /mnt/regress-nfs-client/fstest_7f7ec9d7090c8596dcd634bfe6e258cd:
out of inodes
<3>uid 0 on /mnt/regress-nfs-client/fstest_9e3d690e9879cb6d24f1cba3edcdfa67:
out of inodes
<3>uid 0 on /mnt/regress-nfs-client/fstest_3ba5dcc59f8f6754b05f5a535dcfca98:
out of inodes
<3>uid 0 on /mnt/regress-mfs/fstest_f06ca31319b3edbe10bc4f5956cb576d: file
system full
<3>uid 0 on /mnt/regress-mfs/fstest_8cead83710b5446ba4f1ecd9c29d69bf: out of
inodes
<3>uid 0 on /mnt/regress-mfs/fstest_5cb7431258e75dd3762d2ee7614170b2: out of
inodes
<3>uid 0 on /mnt/regress-mfs/fstest_e85c2954559e68137570ec1316613ff8: out of
inodes
<3>uid 0 on /mnt/regress-mfs/fstest_0fc0633166cc2ba04de32797d262f52d: out of
inodes
msdosfs: reducing max cluster to 129023 from 129025 due to FAT size
<3>thread: table is full
