>Synopsis: page fault trap in rw_status
>Category: kernel
>Environment:
System : OpenBSD 6.9
Details : OpenBSD 6.9 (GENERIC) #4: Mon Jun 7 08:20:14 MDT 2021
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
One of my VMs crashed with a page fault trap
ddb> show panic
kernel page fault
uvm_fault(0xfffffd801a844120, 0x0, 0, 1) -> e
rw_status(0) at rw_status+0x11
end trace frame: 0xffff80000f9991e0, count: 0
ddb> trace
rw_status(0) at rw_status+0x11
amap_wipeout(fffffd8005893740) at amap_wipeout+0x2a
uvm_fault_check(ffff80000f999380,ffff80000f9993b8,ffff80000f9993e0) at uvm_faul
t_check+0x272
uvm_fault(fffffd801a844120,7f7ffffc0000,0,2) at uvm_fault+0xab
upageflttrap(ffff80000f9994e0,7f7ffffc04c8) at upageflttrap+0x59
usertrap(ffff80000f9994e0) at usertrap+0x179
recall_trap() at recall_trap+0x8
end of kernel
end trace frame: 0x7f7ffffc04f0, count: -7
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
24428 68523 86559 1000 2 0x200000 ruby26
14869 509521 1 1000 2 0x200000 ruby26
80979 198985 37543 1000 2 0x200000 ruby26
3035 242284 49553 1000 2 0x200000 ruby26
58747 113954 71576 1000 2 0x200000 ruby26
*86559 462629 6460 1000 7 0x200000 ruby26
6075 220146 1 1000 3 0x82 netio gpg
31153 476748 1 1000 3 0x82 netio gpg
92300 63625 11044 1000 2 0x200002 ruby26
92300 465408 11044 1000 3 0x4200082 poll ruby26
11044 142399 75031 1000 3 0x10008a sigsusp sh
75031 51940 25870 1000 3 0x82 wait mail.mda
25870 45748 46867 1000 3 0x10008a sigsusp sh
54515 175125 1 1000 3 0x82 netio gpg
63292 75610 88483 1000 2 0x200002 ruby26
63292 87569 88483 1000 3 0x4200082 poll ruby26
88483 364345 32794 1000 3 0x10008a sigsusp sh
32794 430523 36517 1000 3 0x82 wait mail.mda
36517 192691 46867 1000 3 0x10008a sigsusp sh
71576 71579 37320 1000 3 0x200082 wait ruby26
71576 468793 37320 1000 3 0x4200082 poll ruby26
37320 333909 83783 1000 3 0x10008a sigsusp sh
83783 507625 1 1000 3 0x82 wait mail.mda
52374 139079 1 1000 3 0x82 netio gpg
80212 84516 70119 1000 2 0x200002 ruby26
80212 376376 70119 1000 2 0x4200002 ruby26
30012 298001 97939 1000 3 0x200082 select ruby26
30012 61319 97939 1000 3 0x4200082 poll ruby26
97939 324099 69895 1000 3 0x10008a sigsusp sh
70119 442064 68687 1000 3 0x10008a sigsusp sh
49553 91029 34056 1000 3 0x200082 wait ruby26
49553 519431 34056 1000 3 0x4200082 poll ruby26
69895 493200 1 1000 3 0x82 wait mail.mda
68687 239578 1 1000 3 0x82 wait mail.mda
34056 427601 63708 1000 3 0x10008a sigsusp sh
70559 284271 1 1000 3 0x82 netio gpg
63708 362934 1 1000 3 0x82 wait mail.mda
6460 119271 30971 1000 3 0x200082 wait ruby26
6460 58664 30971 1000 3 0x4200082 poll ruby26
30971 404502 62816 1000 3 0x10008a sigsusp sh
62816 121173 1 1000 3 0x82 wait mail.mda
78706 369818 1 1000 3 0x82 netio gpg
50900 272995 1 1000 3 0x82 netio gpg
26152 243737 1 1000 3 0x82 netio gpg
37543 299439 46076 1000 3 0x200082 wait ruby26
37543 478159 46076 1000 3 0x4200082 poll ruby26
46076 111965 64614 1000 3 0x10008a sigsusp sh
64614 369138 1 1000 3 0x82 wait mail.mda
46466 118540 31887 1000 3 0x200002 flt_noram1 ruby26
46466 179036 31887 1000 3 0x4200082 poll ruby26
52244 78113 99 1000 3 0x200002 flt_noram1 ruby26
52244 511900 99 1000 3 0x4200082 poll ruby26
31887 159668 94553 1000 3 0x10008a sigsusp sh
94553 441737 1 1000 3 0x82 wait mail.mda
99 124344 93502 1000 3 0x10008a sigsusp sh
93502 23712 1 1000 3 0x82 wait mail.mda
96114 368322 99691 1000 3 0x200002 flt_noram1 ruby26
96114 495837 99691 1000 3 0x4200082 poll ruby26
37570 465739 62218 1000 3 0x200082 select ruby26
37570 181707 62218 1000 3 0x4200082 poll ruby26
99691 181424 58420 1000 3 0x10008a sigsusp sh
62218 437291 79539 1000 3 0x10008a sigsusp sh
58420 117576 1 1000 3 0x82 wait mail.mda
79539 135445 1 1000 3 0x82 wait mail.mda
36579 463221 8205 1000 3 0x200002 flt_noram1 ruby26
36579 493617 8205 1000 3 0x4200082 poll ruby26
8205 244330 30453 1000 3 0x10008a sigsusp sh
30453 278417 1 1000 3 0x82 wait mail.mda
99737 283777 72482 1000 3 0x200002 flt_noram5 ruby26
99737 317179 72482 1000 3 0x4200082 poll ruby26
83659 223936 24172 1000 2 0x200002 ruby26
83659 427905 24172 1000 3 0x4200082 poll ruby26
72482 287385 54162 1000 3 0x10008a sigsusp sh
24172 433726 67728 1000 3 0x10008a sigsusp sh
54162 313170 1 1000 3 0x82 wait mail.mda
67728 53261 1 1000 3 0x82 wait mail.mda
77003 441250 8803 1000 3 0x200002 flt_noram1 ruby26
77003 33884 8803 1000 3 0x4200082 poll ruby26
8803 291869 43935 1000 3 0x10008a sigsusp sh
43935 388312 1 1000 3 0x82 wait mail.mda
25845 328009 82060 1000 3 0x200002 flt_noram1 ruby26
25845 168608 82060 1000 3 0x4200082 poll ruby26
82060 467945 9223 1000 3 0x10008a sigsusp sh
9223 145996 1 1000 3 0x82 wait mail.mda
39936 219339 57707 1000 2 0x200002 ruby26
39936 433626 57707 1000 3 0x4200082 poll ruby26
57707 253964 59197 1000 3 0x10008a sigsusp sh
59197 361773 1 1000 3 0x82 wait mail.mda
63900 98912 85654 1000 3 0x200082 select ruby26
63900 387887 85654 1000 3 0x4200082 poll ruby26
85654 190089 64218 1000 3 0x10008a sigsusp sh
64218 459962 1 1000 3 0x82 wait mail.mda
4785 205086 28954 1000 3 0x200002 flt_noram1 ruby26
4785 31006 28954 1000 3 0x4200082 poll ruby26
28954 73992 88883 1000 3 0x10008a sigsusp sh
88883 74472 1 1000 3 0x82 wait mail.mda
38788 195801 7374 1000 3 0x200082 fsleep ruby26
38788 43833 7374 1000 3 0x4200082 poll ruby26
38788 341383 7374 1000 2 0x4200002 ruby26
7374 462557 8771 1000 3 0x10008a sigsusp sh
8771 500217 1 1000 3 0x82 wait mail.mda
67590 442042 3534 1000 3 0x200082 select ruby26
67590 116313 3534 1000 3 0x4200082 poll ruby26
3534 102216 55240 1000 3 0x10008a sigsusp sh
55240 240312 1 1000 3 0x82 wait mail.mda
9247 99843 79875 1000 3 0x200082 select ruby26
9247 72353 79875 1000 3 0x4200082 poll ruby26
79875 110483 51962 1000 3 0x10008a sigsusp sh
51962 449106 1 1000 3 0x82 wait mail.mda
72109 210917 70621 1000 3 0x200082 select ruby26
72109 22996 70621 1000 3 0x4200082 poll ruby26
70621 395786 65429 1000 3 0x10008a sigsusp sh
65429 300501 1 1000 3 0x82 wait mail.mda
56387 424406 45244 1000 3 0x200082 select ruby26
56387 294601 45244 1000 3 0x4200082 poll ruby26
45244 112953 35402 1000 3 0x10008a sigsusp sh
35402 175028 1 1000 3 0x82 wait mail.mda
81001 116841 52144 1000 3 0x200082 select ruby26
81001 255240 52144 1000 3 0x4200082 poll ruby26
52144 351590 1532 1000 3 0x10008a sigsusp sh
1532 272968 1 1000 3 0x82 wait mail.mda
62142 198491 61461 1000 2 0x200002 ruby26
62142 53338 61461 1000 3 0x4200082 poll ruby26
61461 276474 90429 1000 3 0x10008a sigsusp sh
90429 462566 1 1000 3 0x82 wait mail.mda
83138 284077 71771 1000 2 0x200002 ruby26
83138 475061 71771 1000 3 0x4200082 poll ruby26
71771 186190 66630 1000 3 0x10008a sigsusp sh
66630 521437 92338 0 3 0x10008a sigsusp sh
92338 267257 38350 0 3 0x100082 wait find
38350 364478 66560 0 3 0x10008a sigsusp sh
66560 81992 94967 0 3 0x10008a sigsusp sh
94967 67089 73078 0 3 0x100090 piperd cron
27135 228936 1 1000 3 0x88 select gpg-agent
18157 7948 1 1000 3 0x88 select gpg-agent
58218 438335 1 1000 3 0x88 select gpg-agent
68046 503971 1 1000 2 0 gpg-agent
22243 320109 1 1000 3 0x80 fsleep gpg-agent
22243 152660 1 1000 2 0x4000000 gpg-agent
22243 347752 1 1000 3 0x4000080 fsleep gpg-agent
22243 121584 1 1000 3 0x4000080 fsleep gpg-agent
22243 215949 1 1000 3 0x4000080 fsleep gpg-agent
22243 198275 1 1000 3 0x4000080 fsleep gpg-agent
22243 206581 1 1000 3 0x4000080 fsleep gpg-agent
22243 436666 1 1000 3 0x4000080 fsleep gpg-agent
22243 101378 1 1000 2 0x4000000 gpg-agent
22243 392055 1 1000 3 0x4000080 fsleep gpg-agent
70726 457737 1 0 3 0x100083 ttyin getty
73078 56032 1 0 3 0x100098 poll cron
88626 481742 5798 95 3 0x82 netio table-sqlite
83650 270876 46867 95 3 0x100092 kqread smtpd
83190 436297 46867 103 2 0x100012 smtpd
78472 506926 46867 95 2 0x100012 smtpd
5798 319477 46867 95 3 0x100092 kqread smtpd
12979 12075 46867 95 3 0x100092 kqread smtpd
88920 174682 46867 95 3 0x100092 kqread smtpd
67911 515637 1 1000 3 0x200083 select ruby26
67911 12061 1 1000 3 0x4200083 poll ruby26
67911 309994 1 1000 3 0x4200083 poll ruby26
67911 356333 1 1000 3 0x4200083 fsleep ruby26
67911 244950 1 1000 3 0x4200083 fsleep ruby26
67911 105582 1 1000 3 0x4200083 fsleep ruby26
67911 246016 1 1000 3 0x4200083 fsleep ruby26
67911 131285 1 1000 3 0x4200083 fsleep ruby26
67911 374680 1 1000 3 0x4200083 fsleep ruby26
67911 137977 1 1000 3 0x4200083 fsleep ruby26
67911 304151 1 1000 3 0x4200083 fsleep ruby26
67911 172333 1 1000 3 0x4200083 fsleep ruby26
67911 494760 1 1000 3 0x4200083 fsleep ruby26
67911 372037 1 1000 3 0x4200083 fsleep ruby26
67911 219786 1 1000 3 0x4200083 fsleep ruby26
67911 492925 1 1000 3 0x4200083 fsleep ruby26
67911 202379 1 1000 3 0x4200083 fsleep ruby26
67911 153420 1 1000 3 0x4200083 fsleep ruby26
67911 288625 1 1000 3 0x4200083 fsleep ruby26
67911 490933 1 1000 3 0x4200083 fsleep ruby26
67911 278799 1 1000 3 0x4200083 fsleep ruby26
67911 232499 1 1000 3 0x4200083 fsleep ruby26
46867 496402 1 0 3 0x100080 kqread smtpd
83864 276876 1 0 3 0x80 select sshd
43309 397238 1 0 3 0x100080 poll ntpd
90061 501512 7907 83 3 0x100092 poll ntpd
7907 509192 1 83 2 0x100012 ntpd
51684 386850 72156 74 3 0x100092 bpf pflogd
72156 200779 1 0 3 0x80 netio pflogd
37471 42567 29664 73 3 0x100090 kqread syslogd
29664 409963 1 0 3 0x100082 netio syslogd
73659 473618 1 77 3 0x100090 poll dhclient
40830 304906 1 0 3 0x80 poll dhclient
64082 323050 92829 115 3 0x100092 kqread slaacd
37319 153057 92829 115 3 0x100092 kqread slaacd
92829 521903 1 0 3 0x100080 kqread slaacd
78896 457387 0 0 3 0x14200 bored smr
93652 17849 0 0 3 0x14200 pgzero zerothread
87215 279474 0 0 3 0x14200 aiodoned aiodoned
59661 183077 0 0 3 0x14200 syncer update
77940 205250 0 0 3 0x14200 cleaner cleaner
36255 57846 0 0 3 0x14200 reaper reaper
76648 5714 0 0 2 0x14200 pagedaemon
2392 344045 0 0 3 0x14200 bored crynlk
99148 450806 0 0 3 0x14200 bored crypto
1900 196248 0 0 3 0x14200 bored softnet
43861 293765 0 0 3 0x14200 bored systqmp
82930 430044 0 0 3 0x14200 bored systq
40014 232856 0 0 3 0x40014200 bored softclock
22007 505843 0 0 3 0x40014200 idle0
1 81172 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb> show uvm
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
121478 VM pages: 60248 active, 30019 inactive, 65 wired, 5 free (0 zero)
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
freemin=4049, free-target=5398, inactive-target=30036, wired-max=40492
faults=9051552, traps=9231593, intrs=26202481, ctxswitch=6058861 fpuswitch=0
softint=13077905, syscalls=25514901, kmapent=11
fault counts:
noram=1191470, noanon=0, noamap=0, pgwait=7, pgrele=0
ok relocks(total)=1652265(1652346), anget(retries)=4025885(1468035), amapco
py=1540576
neighbor anon/obj pg=1512054/4165883, gets(lock/unlock)=1652565/184269
cases: anon=3835980, anoncow=189847, obj=1363156, prcopy=288413, przero=337
7927
daemon and swap counts:
woke=1165876, revs=1187026, scans=1832267541, obscans=21522202, anscans=181
0745339
busy=0, freed=746569, reactivate=0, deactivate=1694102
pageouts=94994196, pending=91077, nswget=282429
nswapdev=1
swpages=262143, swpginuse=262142, swpgonly=240501 paging=0
kernel pointers:
objs(kern)=0xffffffff82158318
ddb> show bcstats
Current Buffer Cache status:
numbufs 1461 busymapped 0, delwri 29
kvaslots 1518 avail kva slots 1518
bufpages 6009, dmapages 6009, dirtypages 152
pendingreads 0, pendingwrites 0
highflips 0, highflops 0, dmaflips 0
ddb> dmesg
OpenBSD 6.9 (GENERIC) #4: Mon Jun 7 08:20:14 MDT 2021
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 520081408 (495MB)
avail mem = 489091072 (466MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf36e0 (10 entries)
bios0: vendor SeaBIOS version "1.14.0-OpenBSD-vmm" date 01/01/2011
bios0: OpenBSD VMM
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3401.63 MHz, 06-3a-09
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE
,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,
NXE,LONG,LAHF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: OpenBSD
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio0
virtio0: irq 3
virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address fe:e1:ba:d0:00:04
virtio1: irq 5
virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus1 at vioblk0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 307200MB, 512 bytes/sector, 629145600 sectors
virtio2: irq 6
virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
vmmci0 at virtio3
virtio3: irq 7
isa0 at mainbus0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
com0: console
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (c14ce37920a910f7.a) swap on sd0b dump on sd0b
uvm_fault(0xfffffd801a844120, 0x0, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at rw_status+0x11: movq 0(%rdi),%rcx
