>Synopsis: client hangs accessing nfs server >Category: kernel >Environment: System : OpenBSD 5.9 Details : OpenBSD 5.9-current (GENERIC.MP) #1969: Sun Mar 27 20:58:42 MDT 2016 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64 Machine : amd64 >Description: client hangs after accessing read-only nfs mount on synology server. Pings continue to work. ctl-c won't abort the hung program. Nor will kill -1 or -9. 'shutdown -fr now' from another terminal usually results in 'syncing disks' after a while but never shuts down. A hard reset is required followed by fsck. This is not a new problem. I haven't had time to dig deeper until now. boot sync/crash both panic. While the following info is from the vm instance, the hang occurs w/ real machines in a similar way except I don't have nmi interupt hardware on the bare metal machines to break into ddb. I've appended the ddb results of: trace show registers ps show uvm show bcstats callout show all pools show all mounts show all nfsnodes show all nfsreqs show all nfsnodes /f show all nfsreqs /f show bcstats show extents show malloc boot sync followed by the dmesg. >How-To-Repeat: The problem will repeat immediately as shown in the following transcript. While I haven't tried all combinations, I should note that the hang does *not* occur w/ other OpenBSD and readynas nfs servers nor w/ other mounts on the same or different synology servers. obsd64:build/packages 501>doas mount /nas5/downloads obsd64:build/packages 502>mount -v | grep nfs nas5:/volume1/downloads on /nas5/downloads type nfs (nodev, nosuid, read-only, ctime=Wed Mar 30 18:18:09 2016, v3, udp, hard, wsize=8192, rsize=8192, rdirsize=8192, timeo=100, retrans=101, maxgrouplist=16, readahead=1, acregmin=5, acregmax=60,acdirmin=5, acdirmax=60) obsd64:build/packages 503>df -ih /nas5/downloads Filesystem Size Used Avail Capacity iused ifree %iused Mounted on nas5:/volume1/downloads 21.7T 9.7T 12.1T 44% 3509567 362320577 1% /nas5/downloads obsd64:build/packages 504>ls -l /nas5/downloads/tmp total 3149116 -rw-r--r-- 1 rdo _ldapd 12 Mar 28 15:00 hello_world.txt -rw-rw-r-- 3 rdo _ldapd 3224686592 Jun 1 2015 win7_upgrade.iso obsd64:build/packages 505>dd if=/nas5/downloads/tmp/win7_upgrade.iso of=/dev/null count=1 nfs server nas5:/volume1/downloads: not responding ^G*** FINAL System shutdown message from r...@obsd64.thrush.com ***^G System going down IMMEDIATELY pflogd[33454]: Exiting syncing disks... NMI ... going to debugger >Fix: No corrections known. Eventually all new processes appear to hang requiring a hard reset and fsck. ddb: syncing disks... NMI ... going to debugger Stopped at acpicpu_idle+0xdc: movq 0x60(%r13),%rdx ddb{0}> trace acpicpu_idle() at acpicpu_idle+0xdc cpu_idle_cycle() at cpu_idle_cycle+0x10 end trace frame: 0x0, count: -2 ddb{0}> show registers rdi 0xffffffff81944a80 sched_idle_cpus rsi 0x1 rbp 0xffff800029289ec8 rbx 0x1388 __ALIGN_SIZE+0x388 rdx 0x2710 __ALIGN_SIZE+0x1710 rcx 0 rax 0xffff __ALIGN_SIZE+0xefff r8 0x2 r9 0x2 r10 0x2 r11 0xffffffff8134acb0 x86_bus_space_mem_read_4 r12 0xffff80000004ba40 r13 0xffff800000014e00 r14 0xffffffff818c7420 cpu_info_primary r15 0 rip 0xffffffff816783fc acpicpu_idle+0xdc cs 0x8 rflags 0x297 rsp 0xffff800029289e78 ss 0x10 acpicpu_idle+0xdc: movq 0x60(%r13),%rdx ddb{0}> ps TID PPID PGRP UID S FLAGS WAIT COMMAND 42696 1 42696 0 3 0x2 vinvalbuf reboot 14166 1 13773 0 3 0x2 nfsrcvlk perl 2895 1 2895 1000 3 0x100003 netio dd 80505 0 0 0 3 0x14280 nfsidl nfsio 87500 0 0 0 3 0x14280 nfsidl nfsio 94937 0 0 0 3 0x14280 nfsidl nfsio 12900 0 0 0 3 0x14200 nfsrcvlk nfsio 60578 0 0 0 3 0x14200 pgzero zerothread 55634 0 0 0 3 0x14200 aiodoned aiodoned 93109 0 0 0 3 0x14200 syncer update 93380 0 0 0 3 0x14200 cleaner cleaner 3347 0 0 0 3 0x14200 reaper reaper 1838 0 0 0 3 0x14200 pgdaemon pagedaemon 57000 0 0 0 3 0x14200 bored crynlk 76542 0 0 0 3 0x14200 bored crypto 30890 0 0 0 3 0x14200 pftm pfpurge 95241 0 0 0 3 0x14200 usbtsk usbtask 83914 0 0 0 3 0x14200 usbatsk usbatsk 16820 0 0 0 3 0x14200 bored viomb 13815 0 0 0 3 0x40014200 acpi0 acpi0 51429 0 0 0 7 0x40014200 idle1 63826 0 0 0 3 0x14200 bored softnet 3248 0 0 0 3 0x14200 bored systqmp 60104 0 0 0 3 0x14200 bored systq *15915 0 0 0 7 0x40014200 idle0 27812 0 0 0 3 0x14200 bored sbar 1 0 1 0 3 0x82 wait init 0 -1 0 0 3 0x10200 scheduler swapper ddb{0}> show uvm Current UVM status: pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12 505452 VM pages: 2863 active, 3060 inactive, 1 wired, 465828 free (58231 zero ) min 10% (25) anon, 10% (25) vnode, 5% (12) vtext pages 0 anon, 0 vnode, 0 vtext freemin=16848, free-target=22464, inactive-target=0, wired-max=168484 faults=568530, traps=327746, intrs=46352, ctxswitch=876522 fpuswitch=712 softint=6139015, syscalls=1809376, kmapent=18 fault counts: noram=0, noanon=0, pgwait=0, pgrele=0 ok relocks(total)=3766(3768), anget(retries)=436157(0), amapcopy=155704 neighbor anon/obj pg=23655/170507, gets(lock/unlock)=56173/3768 cases: anon=394283, anoncow=41874, obj=48246, prcopy=7925, przero=76147 daemon and swap counts: woke=0, revs=0, scans=0, obscans=0, anscans=0 busy=0, freed=0, reactivate=0, deactivate=0 pageouts=0, pending=0, nswget=0 nswapdev=1, nanon=0, nanonneeded=0 nfreeanon=0 swpages=585695, swpginuse=0, swpgonly=0 paging=0 kernel pointers: objs(kern)=0xffffffff81909860 ddb{0}> show bcstats Current Buffer Cache status: numbufs 6724 busymapped 2, delwri 0 kvaslots 6318 avail kva slots 6316 bufpages 26857, dirtypages 0 pendingreads 2, pendingwrites 0 ddb{0}> callout ticks now: -2141447390 ticks wheel arg func 1 0d/35 d 0xffff800000384800 nfs_timer 1 0d/35 d 0xffffffff8194e440 net_tick 2 0d/36 d 0xffffffff818e93a0 pffasttimo 18 0d/52 d 0xffff80000006c000 uhci_poll_hub 41 0d/75 d 0xffffffff818e9580 rt_timer_timer 41 0d/75 d 0xffffffff818e8fe0 schedcpu 41 0d/75 d 0x0 nd6_timer 41 0d/75 d 0xffff800029275868 endtsleep 41 0d/75 d 0x0 pool_gc_sched 42 0d/76 d 0xffffffff81937b60 pckbc_poll 42 0d/76 d 0xffffffff818e9360 pfslowtimo 424 1d/284 d 0xffff8000292f6248 endtsleep 26594 1d/387 d 0xffffffff818e95c0 arptimer 53742 1d/493 d 0x0 arc4_reinit 353741 2d/609 d 0x0 nd6_slowtimo ddb{0}> show all pools Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle arp 56 225 0 219 1 0 1 1 0 8 0 inpcbpl 288 416 0 415 134 133 1 2 0 8 0 plimitpl 152 59 0 56 2 1 1 1 0 8 0 rtentry 112 249 0 219 1 0 1 1 0 8 0 syncache 264 1 0 1 1 1 0 1 0 8 0 tcpqe 32 2 0 2 2 2 0 1 0 8 0 tcpcb 560 129 0 129 1 1 0 1 0 8 0 art_node 24 249 0 219 1 0 1 1 0 8 0 art_heap8 4096 1 0 0 1 0 1 1 0 8 0 art_heap4 256 122 0 22 7 0 7 7 0 8 0 art_table 32 123 0 22 1 0 1 1 0 8 0 pfosfp 40 840 0 420 5 0 5 5 0 8 0 pfosfpen 112 1420 0 710 21 0 21 21 0 8 0 pfstitem 24 3119 0 3118 1 0 1 1 0 8 0 pfstkey 104 3119 0 3118 3 2 1 3 0 8 0 pfstate 312 3119 0 3118 11 10 1 9 0 8 0 pfrule 1336 19 0 16 2 1 1 2 0 8 0 nfsnodepl 472 5 0 4 1 0 1 1 0 8 0 dirhash 1024 674 0 13 83 0 83 83 0 8 0 dino1pl 128 55858 0 49211 215 0 215 215 0 8 0 ffsino 240 55858 0 49211 392 0 392 392 0 8 0 nchpl 144 63602 0 58324 196 0 196 196 0 8 0 uvmvnodes 72 6664 0 0 122 0 122 122 0 8 0 vnodes 200 6664 0 0 351 0 351 351 0 8 0 namei 1024 110706 0 110706 1079 1079 0 2 0 8 0 nfsreqpl 96 16 0 13 3 2 1 1 0 8 0 uhcixfer 264 86 0 85 1 0 1 1 0 8 0 wdcxfer 176 9 0 9 1 1 0 1 0 8 0 scxspl 192 28141 0 28141 1554 1554 0 4 0 8 0 sigapl 432 1146 0 1142 8 4 4 5 0 8 0 knotepl 112 584 0 584 2 2 0 2 0 8 0 kqueuepl 320 8 0 8 1 1 0 1 0 8 0 pipepl 120 1120 0 1116 10 9 1 1 0 8 0 fdescpl 440 1147 0 1142 6 3 3 5 0 8 0 filepl 112 15465 0 15457 4 1 3 4 0 8 0 lockfpl 88 78 0 78 2 2 0 1 0 8 0 sessionpl 64 46 0 42 1 0 1 1 0 8 0 pgrppl 40 60 0 56 1 0 1 1 0 8 0 ucredpl 96 841 0 835 1 0 1 1 0 8 0 zombiepl 144 1142 0 1142 93 93 0 1 0 8 0 processpl 672 1169 0 1142 10 6 4 6 0 8 0 procpl 568 1169 0 1142 6 1 5 5 0 8 0 sockpl 392 679 0 678 10 9 1 9 0 8 0 mcl2k 2048 16954 0 16927 6 1 5 6 1 768 1 mtagpl 72 1 0 1 1 1 0 1 0 8 0 mbufpl 256 24506 0 24469 6 1 5 5 1 384 1 bufpl 256 24973 0 18249 421 0 421 421 0 8 0 anonpl 16 125946 0 123776 40 21 19 36 0 127 0 amapslotpl16 256 486 0 475 7 5 2 3 0 256 0 amapslotpl15 240 679 0 678 4 3 1 2 0 241 0 amapslotpl14 224 60 0 60 1 1 0 1 0 241 0 amapslotpl13 208 691 0 690 1 0 1 1 0 216 0 amapslotpl12 192 113 0 113 7 7 0 1 0 205 0 amapslotpl11 176 398 0 396 4 3 1 4 0 187 0 amapslotpl10 160 1838 0 1836 3 2 1 2 0 171 0 amapslotpl9 144 252 0 251 1 0 1 1 0 152 0 amapslotpl8 128 185 0 184 2 1 1 1 0 133 0 amapslotpl7 112 868 0 864 1 0 1 1 0 118 0 amapslotpl6 96 245 0 245 2 2 0 1 0 100 0 amapslotpl5 80 468 0 468 3 3 0 2 0 84 0 amapslotpl4 64 2736 0 2522 17 11 6 6 0 67 0 amapslotpl3 48 1895 0 1822 4 2 2 2 0 50 0 amapslotpl2 32 9159 0 9062 5 3 2 5 0 34 0 amapslotpl1 16 92727 0 91907 20 11 9 20 0 17 0 amappl 72 113381 0 112154 156 111 45 112 0 75 0 dma4096 4096 2 0 2 1 1 0 1 0 8 0 dma1024 1024 1 0 0 1 0 1 1 0 8 0 dma512 512 2 0 2 1 1 0 1 0 8 0 dma256 256 16 0 16 3 3 0 1 0 8 0 dma64 64 7 0 7 1 1 0 1 0 8 0 dma32 32 14 0 14 3 3 0 1 0 8 0 aobjpl 64 1 0 0 1 0 1 1 0 8 0 uaddrrnd 24 1147 0 1142 1 0 1 1 0 8 0 uaddrbest 32 2 0 0 1 0 1 1 0 8 0 uaddr 24 1147 0 1142 1 0 1 1 0 8 0 vmmpekpl 168 51749 0 51723 2 0 2 2 0 8 0 vmmpepl 168 400864 0 399497 835 759 76 506 0 357 0 vmsppl 256 1146 0 1142 4 1 3 3 0 8 0 pdppl 4096 1146 0 1142 7 4 3 6 0 8 0 pvpl 32 772363 0 769324 334 286 48 138 0 265 0 pmappl 176 1146 0 1142 2 0 2 2 0 8 0 extentpl 40 60 0 34 1 0 1 1 0 8 0 phpool 112 2172 0 1242 27 0 27 27 0 8 0 ddb{0}> show all mounts flags 50000<LOCAL,ROOTFS> vnodecovered 0x0 syncer 0xffffff006fe60400 data 0xffff8000000d9a00 vfsconf: ops 0xffffffff817361a0 name "ffs" num 1 ref 6 flags 0x1000 statvfs cache: bsize 800 iosize 4000 blocks 514439 free 445332 avail 419611 files 155902 ffiles 150203 favail 150203 f_fsidx {0x400, 0x60be41b4} owner 0 ctime 0x56fc502b syncwrites 403 asyncwrites = 251 syncreads 37417 asyncreads = 0 fstype "ffs" mnton "/" mntfrom "/dev/sd0a" mntspec "8f3e304cddb66a7a.a" locked vnodes: flags 10030<NOSUID,NODEV,LOCAL> vnodecovered 0xffffff006fe604c8 syncer 0xffffff006fe60590 data 0xffff800000197e 00 vfsconf: ops 0xffffffff817361a0 name "ffs" num 1 ref 6 flags 0x1000 statvfs cache: bsize 800 iosize 4000 blocks 252695 free 173187 avail 160553 files 64510 ffiles 58466 favail 58466 f_fsidx {0x407, 0xcfb167c2} owner 0 ctime 0x56fc5028 syncwrites 21 asyncwrites = 57 syncreads 15177 asyncreads = 0 fstype "ffs" mnton "/home" mntfrom "/dev/sd0h" mntspec "8f3e304cddb66a7a.h" locked vnodes: flags 10021<RDONLY,NODEV,LOCAL> vnodecovered 0xffffff007e600718 syncer 0x0 data 0xffff800000197200 vfsconf: ops 0xffffffff817361a0 name "ffs" num 1 ref 6 flags 0x1000 statvfs cache: bsize 800 iosize 4000 blocks 1030527 free 657467 avail 605941 files 285822 ffiles 261664 favail 261664 f_fsidx {0x405, 0xe953b157} owner 0 ctime 0x56fc5028 syncwrites 0 asyncwrites = 0 syncreads 67081 asyncreads = 0 fstype "ffs" mnton "/usr" mntfrom "/dev/sd0f" mntspec "8f3e304cddb66a7a.f" locked vnodes: flags 10021<RDONLY,NODEV,LOCAL> vnodecovered 0xffffff006fe60720 syncer 0x0 data 0xffff800000197600 vfsconf: ops 0xffffffff817361a0 name "ffs" num 1 ref 6 flags 0x1000 statvfs cache: bsize 800 iosize 4000 blocks 8161911 free 7996295 avail 7588200 files 2078718 ffiles 2068352 favail 2068352 f_fsidx {0x40b, 0x3922e8d0} owner 0 ctime 0x56fc502f syncwrites 4 asyncwrites = 1 syncreads 22364 asyncreads = 0 fstype "ffs" mnton "/usr/local" mntfrom "/dev/sd0l" mntspec "/dev/sd0l" locked vnodes: flags 10130<NOSUID,NODEV,ASYNC,LOCAL> vnodecovered 0xffffff006fe608b0 syncer 0xffffff006fe60978 data 0xffff800000197a 00 vfsconf: ops 0xffffffff817361a0 name "ffs" num 1 ref 6 flags 0x1000 statvfs cache: bsize 800 iosize 4000 blocks 25799454 free 24486298 avail 23196326 files 6573950 ffiles 6567547 favail 6567547 f_fsidx {0x41e, 0x55c870a7} owner 0 ctime 0x56fc5028 syncwrites 0 asyncwrites = 21 syncreads 13748 asyncreads = 0 fstype "ffs" mnton "/usr/obj" mntfrom "/dev/sd1o" mntspec "c1a908809de1d866.o " locked vnodes: flags 10030<NOSUID,NODEV,LOCAL> vnodecovered 0xffffff006fe60a40 syncer 0xffffff006fe60b08 data 0xffff8000001668 00 vfsconf: ops 0xffffffff817361a0 name "ffs" num 1 ref 6 flags 0x1000 statvfs cache: bsize 800 iosize 4000 blocks 1716823 free 1700279 avail 1614438 files 441726 ffiles 440023 favail 440023 f_fsidx {0x404, 0x1dd6e3be} owner 0 ctime 0x56fc5028 syncwrites 588 asyncwrites = 2134 syncreads 15893 asyncreads = 0 fstype "ffs" mnton "/var" mntfrom "/dev/sd0e" mntspec "8f3e304cddb66a7a.e" locked vnodes: flags 31<RDONLY,NOSUID,NODEV> vnodecovered 0xffffff006bd40d70 syncer 0x0 data 0xffff800000384800 vfsconf: ops 0xffffffff81734da0 name "nfs" num 2 ref 1 flags 0x0 statvfs cache: bsize 200 iosize 2000 blocks 46640139576 free 25910615264 avail 25910377696 files 365830144 ffiles 362320577 favail 362320577 f_fsidx {0x1601, 0x2} owner 0 ctime 0x56fc50a1 syncwrites 0 asyncwrites = 0 syncreads 0 asyncreads = 0 fstype "nfs" mnton "/nas5/downloads" mntfrom "nas5:/volume1/downloads" mntspe c "nas5:/volume1/downloads" locked vnodes: ddb{0}> show all nfsnodes size 3224686592 flag 0 vnode 0xffffff007f3d5288 accstamp 1459376319 ddb{0}> show all nfsreqs xid 0x2a1000f8 flags 0x22 rexmit 100 procnum 6 proc 0xffff8000292f6248 xid 0x1c36cacf flags 0x22 rexmit 100 procnum 6 proc 0xffff8000293786c0 xid 0x127a947 flags 0x1 rexmit 0 procnum 1 proc 0xffff8000293a0260 ddb{0}> show all nfsnodes /f size 3224686592 flag 0 vnode 0xffffff007f3d5288 accstamp 1459376319 pushedlo 0 pushedhi 0 pushlo 0 pushhi 0 commitflags 0 ddb{0}> show all nfsreqs /f xid 0x2a1000f8 flags 0x22 rexmit 100 procnum 6 proc 0xffff8000292f6248 mreq 0xffffffff8173462d mrep 0xffffff007f797e00 md 0x0 nfsmount 0xffffff006f67e 800 vnode 0xffff800000384800 timer 2134725256xid 0x1c36cacf flags 0x22 rexmit 1 00 procnum 6 proc 0xffff8000293786c0 mreq 0xffffffff8173462d mrep 0xffffff006f67ce00 md 0x0 nfsmount 0x0 vnode 0xfff f800000384800 timer 2134725256xid 0x127a947 flags 0x1 rexmit 0 procnum 1 proc 0 xffff8000293a0260 mreq 0xffffffff8173462d mrep 0xffffff006b72ad00 md 0x0 nfsmount 0x207678722d206 464 vnode 0xffff800000384800 timer 1729678760 ddb{0}> show bcstats Current Buffer Cache status: numbufs 6724 busymapped 2, delwri 0 kvaslots 6318 avail kva slots 6316 bufpages 26857, dirtypages 0 pendingreads 2, pendingwrites 0 ddb{0}> show extents extent `swap0x0000' (0x0 - 0x8efdf), flags=0 0x0 - 0x0 0x8efde - 0x8efde extent `swapmap' (0x1 - 0x7fffffff), flags=0 0x1 - 0x8efe0 extent `pcibus' (0x0 - 0xff), flags=0 0x0 - 0x0 extent `pcimem' (0x0 - 0xffffffffffffffff), flags=0 0x0 - 0x7fffffff 0xfc000000 - 0xfdffffff 0xfebf0000 - 0xfebf3fff 0xfeffc000 - 0xfeffffff 0xfffc0000 - 0xffffffff 0x40000000000 - 0xffffffffffffffff extent `pciio' (0x0 - 0xffffffff), flags=0 0xc000 - 0xc0ef 0x10000 - 0xffffffff extent `iomem' (0x0 - 0xffffffffffff), flags=3<NOCOALESCE,FIXED> 0x23000 - 0x9efff 0xa0000 - 0xbffff 0x100000 - 0xffffff 0x1000000 - 0x7ffdffff 0xfed00000 - 0xfed003ff extent `ioport' (0x0 - 0xffff), flags=3<NOCOALESCE,FIXED> 0x0 - 0xb 0x40 - 0x43 0x60 - 0x60 0x61 - 0x61 0x64 - 0x64 0x80 - 0x8e 0xb2 - 0xb2 0xc0 - 0xd7 0x170 - 0x177 0x1f0 - 0x1f7 0x376 - 0x376 0x3c0 - 0x3cf 0x3d0 - 0x3df 0x3f0 - 0x3f5 0x3f6 - 0x3f6 0x3f7 - 0x3f7 0x3f8 - 0x3ff 0x600 - 0x601 0x602 - 0x603 0x604 - 0x605 0x608 - 0x60b 0x700 - 0x70f 0xafe0 - 0xafe1 0xafe2 - 0xafe3 0xc000 - 0xc03f 0xc040 - 0xc07f 0xc080 - 0xc09f 0xc0a0 - 0xc0bf 0xc0c0 - 0xc0df 0xc0e0 - 0xc0ef ddb{0}> show malloc Type InUse MemUse HighUse Limit Requests Type Lim Kern Lim devbuf 1596 834K 964K 157286K 2526 0 0 pcb 13 8K 16K 157286K 276 0 0 rtable 78 2K 2K 157286K 572 0 0 ifaddr 25 7K 7K 157286K 26 0 0 sysctl 3 1K 1K 157286K 3 0 0 ioctlops 0 0K 4K 157286K 1698 0 0 mount 7 7K 7K 157286K 7 0 0 NFS mount 1 0K 0K 157286K 1 0 0 vnodes 27 1K 77K 157286K 2408 0 0 UFS quota 1 32K 32K 157286K 1 0 0 UFS mount 23 61K 61K 157286K 24 0 0 shm 2 1K 1K 157286K 2 0 0 VM map 2 0K 0K 157286K 2 0 0 sem 2 0K 0K 157286K 2 0 0 dirhash 255 54K 54K 157286K 261 0 0 ACPI 867 98K 279K 157286K 19197 0 0 file desc 1 4K 16K 157286K 30 0 0 proc 15 9K 9K 157286K 15 0 0 NFS srvsock 1 0K 0K 157286K 1 0 0 NFS daemon 1 16K 16K 157286K 1 0 0 in_multi 14 0K 0K 157286K 14 0 0 ether_multi 1 0K 0K 157286K 1 0 0 ISOFS mount 1 32K 32K 157286K 1 0 0 MSDOSFS mount 1 16K 16K 157286K 1 0 0 ttys 414 1750K 1750K 157286K 414 0 0 exec 0 0K 2K 157286K 1576 0 0 pfkey data 1 0K 0K 157286K 2 0 0 pagedep 1 8K 8K 157286K 1 0 0 inodedep 1 32K 32K 157286K 1 0 0 newblk 1 0K 0K 157286K 1 0 0 VM swap 7 166K 166K 157286K 7 0 0 UVM amap 16 0K 60K 157286K 2046 0 0 UVM aobj 2 2K 2K 157286K 2 0 0 USB 20 2K 2K 157286K 21 0 0 USB device 5 0K 0K 157286K 5 0 0 memdesc 1 4K 4K 157286K 1 0 0 crypto data 1 1K 1K 157286K 1 0 0 NDP 4 0K 0K 157286K 4 0 0 temp 48 18K 170K 157286K 18481 0 0 ddb{0}> boot sync panic: kernel diagnostic assertion "_kernel_lock_held()" failed: file "../../../../net/rtable.c", line 839 Stopped at Debugger+0x9: leave TID PID UID PRFLAGS PFLAGS CPU COMMAND Debugger() at Debugger+0x9 panic() at panic+0xfe __assert() at __assert+0x25 rtable_walk_helper() at rtable_walk_helper+0x87 art_walk() at art_walk+0x3e rtable_walk() at rtable_walk+0x5d rt_if_track() at rt_if_track+0x74 if_linkstate() at if_linkstate+0x30 if_downall() at if_downall+0x5a boot() at boot+0xe4 reboot() at reboot+0x26 db_boot_sync_cmd() at db_boot_sync_cmd+0x12 db_command() at db_command+0x13a db_command_loop() at db_command_loop+0xb0 end trace frame: 0xffff800029289cc0, count: 0 http://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. >> OpenBSD/amd64 BOOT 3.30 Disk BIOS# Type Cyls Heads Secs Flags Checksum hd0 0x80 label 1023 255 63 0x2 0xdce59776 hd1 0x81 label 1023 255 63 0x2 0x2db005d6 Region 0: type 1 at 0x0 for 639KB Region 1: type 2 at 0x9fc00 for 1KB Region 2: type 2 at 0xf0000 for 64KB Region 3: type 1 at 0x100000 for 2096000KB Region 4: type 2 at 0x7ffe0000 for 128KB Region 5: type 2 at 0xfeffc000 for 16KB Region 6: type 2 at 0xfffc0000 for 256KB Low ram: 639KB High ram: 2096000KB Total free memory: 2096639KB boot> dmesg: OpenBSD 5.9-current (GENERIC.MP) #1969: Sun Mar 27 20:58:42 MDT 2016 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2130575360 (2031MB) avail mem = 2061713408 (1966MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf0cd0 (9 entries) bios0: vendor SeaBIOS version "rel-1.7.5.1-0-g8936dbb-20141113_115728-nilsson.home.kraxel.org" date 04/01/2014 bios0: QEMU Standard PC (i440FX + PIIX, 1996) acpi0 at bios0: rev 0 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP SSDT APIC HPET acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Common KVM processor, 3400.50 MHz cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 1000MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Common KVM processor, 3400.14 MHz cpu1: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins acpihpet0 at acpi0: 100000000 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: KVM pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility pciide0: channel 0 disabled (no drives) atapiscsi0 at pciide0 channel 1 drive 0 scsibus1 at atapiscsi0: 2 targets cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.2.> ATAPI 5/cdrom removable cd0(pciide0:1:0): using PIO mode 4, DMA mode 2 uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11 piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9 iic0 at piixpm0 vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Memory" rev 0x00 viomb0 at virtio0 virtio0: apic 0 int 11 virtio1 at pci0 dev 10 function 0 "Qumranet Virtio Storage" rev 0x00 vioblk0 at virtio1 scsibus2 at vioblk0: 2 targets sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed sd0: 32768MB, 512 bytes/sector, 67108864 sectors virtio1: apic 0 int 10 virtio2 at pci0 dev 11 function 0 "Qumranet Virtio Storage" rev 0x00 vioblk1 at virtio2 scsibus3 at vioblk1: 2 targets sd1 at scsibus3 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed sd1: 51200MB, 512 bytes/sector, 104857600 sectors virtio2: apic 0 int 11 virtio3 at pci0 dev 18 function 0 "Qumranet Virtio Network" rev 0x00 vio0 at virtio3: address 8a:2e:d1:64:f7:6b virtio3: apic 0 int 10 isa0 at pcib0 isadma0 at isa0 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 1: density unknown com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 usb0 at uhci0: USB revision 1.0 uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 nvram: invalid checksum uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2 uhidev0: iclass 3/0 ums0 at uhidev0: 3 buttons, Z dir wsmouse1 at ums0 mux 0 vscsi0 at root scsibus4 at vscsi0: 256 targets softraid0 at root scsibus5 at softraid0: 256 targets root on sd0a (8f3e304cddb66a7a.a) swap on sd0b dump on sd0b WARNING: / was not properly unmounted clock: unknown CMOS layout usbdevs: Controller /dev/usb0: addr 1: full speed, self powered, config 1, UHCI root hub(0x0000), Intel(0x8086), rev 1.00 port 1 addr 2: full speed, power 100 mA, config 1, QEMU USB Tablet(0x0001), QEMU(0x0627), rev 0.00, iSerialNumber 42 port 2 powered