I tried building a kernel with LOCKDEBUG, but that hung about halfway through the /etc/rc.d/ scripts, so I gave up on it. I've just managed to get a proper core dump from a hang. Didn't write down the backtraces for the various CPUs, but one was doing disk I/O, one was in pf handling a fragment (and calling kernel_lock()), one was in ipmi (again, calling kernel_lock(), I think), and the final one was servicing some interrupt.
If there's anything I ought to pull from the dump, please let me know! Here's what it looks like (there's a full build running): barsoom# /usr/src/usr.sbin/crash/obj.amd64/crash -M netbsd.27.core -N /netbsd Crash version 7.99.4, image version 7.99.4. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. crash> ps PID LID S CPU FLAGS STRUCT LWP * NAME WAIT 8212 1 2 0 8020000 fffffe81d63c5a00 arp 20073 1 2 0 8020000 fffffe810922b040 sh 11360 1 2 0 8020000 fffffe81b1ee0340 pickup 17479 1 3 3 8060000 fffffe81d62c49c0 nbmake biolock 14776 1 2 0 8020000 fffffe8120ce0420 sh 6231 1 3 1 8020000 fffffe81ef6f44a0 as biolock 26596 1 2 0 8020000 fffffe810b26b020 x86_64--netbsd-g 9704 1 2 0 8020000 fffffe81e8b5c540 sh 13942 1 2 3 8020000 fffffe81c4f59ae0 nbmake 13307 1 2 0 8020000 fffffe81ecc9f8a0 sh 12557 1 3 2 8020000 fffffe8198bed260 as biolock 5203 1 2 0 8020000 fffffe81ea25eb40 cc1 620 1 2 0 8020000 fffffe81c1ab4740 as 13884> 1 7 0 8020000 fffffe81ccd42a60 cc1plus 16491 1 2 0 8020000 fffffe81e69bdb00 x86_64--netbsd-g 8613 1 2 0 8020000 fffffe81e7c204c0 x86_64--netbsd-c 16001 1 2 0 8020000 fffffe8226b83020 sh 565 1 2 0 8020000 fffffe81c51c0780 sh 3687 1 2 0 8020000 fffffe81ebce66c0 nbmake 16941 1 2 0 8020000 fffffe81ebb9e000 sh 15688 1 2 0 8020000 fffffe81e91a62e0 nbmake 18200 1 2 0 8020000 fffffe81d2542520 sh 7642 1 2 0 8020000 fffffe81ec09f320 nbmake 15077 1 2 0 8020000 fffffe8127f464a0 sh 4055 1 2 0 8020000 fffffe81e69bd2c0 nbmake 11510 1 2 0 8020000 fffffe8198bedaa0 sh 17395 1 2 0 8020000 fffffe81d63051a0 nbmake 842 1 2 0 8020000 fffffe810922b460 sh 20753 1 2 0 8020000 fffffe81e8b5c120 nbmake 2227 1 2 0 8020000 fffffe81d2542100 sh 17156 1 2 3 8020000 fffffe81d63c55e0 nbmake 9219 1 2 0 8020000 fffffe81c30e1b20 sh 20959 1 2 0 8020000 fffffe8236560980 nbmake 2929 1 2 0 8020000 fffffe81d52041e0 sh 21452 1 2 0 8020000 fffffe81cbb42a80 nbmake 15300 1 2 0 8020000 fffffe81ccd42220 sh 29217 1 2 0 8020000 fffffe81ead53280 nbmake 7574 1 2 0 8020000 fffffe81c30e1700 sh 18800 1 2 0 8020000 fffffe81ee173040 nbmake 10467 1 2 0 8020000 fffffe81e9e20020 sh 23887 1 2 0 8020000 fffffe81b1ee0760 nbmake 5297 1 2 0 8020000 fffffe81d2542940 sh 13346 1 2 3 8020000 fffffe81ef6f4080 nbmake 21577 1 2 0 8020000 fffffe8127f468c0 sh 29384 1 2 0 8020000 fffffe81e7a1e500 nbmake 24009 1 2 0 8020000 fffffe81ea25e300 sh 12390 1 2 0 8020000 fffffe81ea25e720 nbmake 17723 1 2 0 8020000 fffffe81d16c2140 sh 28866 1 2 3 8020000 fffffe823d8696c0 nbmake 8504 1 2 0 8020000 fffffe8198bed680 sh 21990 1 2 0 8020000 fffffe81c53abb00 nbmake 3246 1 2 0 8020000 fffffe81ebb9e840 sh 7389 1 2 3 8020000 fffffe81ec09f740 nbmake 28730 1 2 0 8020000 fffffe81d63055c0 sh 5272 1 2 0 8020000 fffffe81c30e12e0 anvil 13713 1 2 0 8020000 fffffe81ebedfb80 nbmake 28165 1 2 0 8020000 fffffe81c53ab6e0 sh 12995 1 2 3 8020000 fffffe81ead536a0 nbmake 12934 1 2 0 8020000 fffffe81cc6c2200 sh 2973 1 2 0 8020000 fffffe822c8c3880 nbmake 9325 1 2 0 8020000 fffffe81e7c200a0 grep 7904 1 2 0 8020000 fffffe81ef6f48c0 tail 6440 1 2 0 8020000 fffffe8230454760 sh 6693 1 2 3 8020000 fffffe8233f000e0 nbmake 7997 1 2 0 8020000 fffffe823d5da6a0 sh 6671 1 2 0 8020000 fffffe823d5daac0 nbmake 6074 1 2 0 8020000 fffffe823d5da280 sh 3467 1 2 3 8020000 fffffe823d3456e0 nbmake 2095 1 2 0 8020000 fffffe823d345b00 sh 2774 1 2 0 8020000 fffffe8230d53660 sh 3285 1 2 0 8020000 fffffe81ed561680 bash 2771 1 2 0 8020000 fffffe81ed561aa0 sshd 2444 1 2 0 8020000 fffffe8230d53240 sshd 1460 1 2 0 8020000 fffffe8230d53a80 sh 2111 1 2 0 8020000 fffffe8234da1220 su 3894 1 2 0 8020000 fffffe8234da1640 bash 1088 1 2 0 8020000 fffffe8234da1a60 sshd 2029 1 2 0 8020000 fffffe81ef2c9200 sshd 3317 1 2 0 8020000 fffffe823d75b260 getty 2028 37 2 0 8020000 fffffe81ef2c9620 asterisk 2028 36 2 0 8020000 fffffe823d869ae0 asterisk 2028 35 2 0 8020000 fffffe823330b9e0 asterisk 2028 34 2 0 8020000 fffffe81ef2c9a40 asterisk 2028 33 2 0 8020000 fffffe81d26001e0 asterisk 2028 32 2 1 8020000 fffffe81d2600600 asterisk 2028 31 2 0 8020000 fffffe81d2600a20 asterisk 2028 30 2 1 8020000 fffffe81e4b031c0 asterisk 2028 29 2 0 8020000 fffffe81e4b035e0 asterisk 2028 28 2 0 8020000 fffffe81e4b03a00 asterisk 2028 27 2 0 8020000 fffffe81d2b001a0 asterisk 2028 26 2 0 8020000 fffffe81d2b005c0 asterisk 2028 25 2 0 8020000 fffffe81d2b009e0 asterisk 2028 24 2 0 8020000 fffffe81d2bc0180 asterisk 2028 23 2 0 8020000 fffffe81d2bc05a0 asterisk 2028 22 2 0 8020000 fffffe81d2bc09c0 asterisk 2028 21 2 0 8020000 fffffe81d2c80160 asterisk 2028 20 2 0 8020000 fffffe81d2c80580 asterisk 2028 19 2 0 8020000 fffffe81d2c809a0 asterisk 2028 18 2 0 8020000 fffffe81d2d40140 asterisk 2028 17 2 0 8020000 fffffe81d2d40560 asterisk 2028 16 2 0 8020000 fffffe81d2d40980 asterisk 2028 15 2 0 8020000 fffffe81d37c0120 asterisk 2028 14 2 1 8020000 fffffe81d37c0540 asterisk 2028 13 2 0 8020000 fffffe81d37c0960 asterisk 2028 12 2 0 8020000 fffffe81d6b80100 asterisk 2028 11 2 0 8020000 fffffe81d6b80520 asterisk 2028 10 2 0 8020000 fffffe81d6b80940 asterisk 2028 9 2 0 8020000 fffffe820ea400e0 asterisk 2028 8 2 0 8020000 fffffe820ea40500 asterisk 2028 7 2 0 8020000 fffffe820ea40920 asterisk 2028 6 2 0 8020000 fffffe81f519e0c0 asterisk 2028 5 2 0 8020000 fffffe823d8692a0 asterisk 2028 4 2 0 8020000 fffffe81f519e4e0 asterisk 2028 3 2 0 8020000 fffffe81f519e900 asterisk 2028 2 2 0 8020000 fffffe823cead740 asterisk 2028 1 2 0 8020000 fffffe81f3c040a0 asterisk 225 2 2 1 8020000 fffffe823330b1a0 bacula-fd 225 1 2 0 8020000 fffffe823ceadb60 bacula-fd 3405 1 2 0 8020000 fffffe81f3c044c0 cupsd 2238 1 2 0 8020000 fffffe81fa3fd4a0 dbus-daemon 3310 1 2 0 8020000 fffffe82320ee640 python2.7 2664 1 2 1 8020000 fffffe82320eea60 python2.7 1695 1 2 1 8020000 fffffe822ca42660 python2.7 1980 1 2 1 8020000 fffffe822ca42a80 python2.7 1770 1 2 1 8020000 fffffe822c6c2680 python2.7 1769 1 2 1 8020000 fffffe822c6c2260 python2.7 2660 1 2 1 8020000 fffffe822ca42240 python2.7 929 1 2 1 8020000 fffffe822c6c2aa0 python2.7 2587 1 2 0 8020000 fffffe81f3c048e0 python2.7 2512 1 2 0 8020000 fffffe81fa3fd080 python2.7 2361 9 5 0 8020000 fffffe8234c1d5e0 mfschunkserver 2361 8 2 0 9020000 fffffe81fa3fd8c0 mfschunkserver 2361 7 2 1 9020000 fffffe81e3340060 mfschunkserver 2361 6 2 0 9020000 fffffe81e3340480 mfschunkserver 2361 > 5 7 1 9020000 fffffe822da6e840 mfschunkserver 2361 4 2 0 9020000 fffffe81e33408a0 mfschunkserver 2361 3 2 0 9020000 fffffe822da6e000 mfschunkserver 2361 2 2 0 9020000 fffffe822da6e420 mfschunkserver 2361 1 2 0 9020000 fffffe8226b83860 mfschunkserver 1916 1 2 3 8020000 fffffe8226b83440 mfsmaster 1827 1 2 3 8020000 fffffe822d84f360 perl 409 1 2 0 8020000 fffffe8228303740 nmbd 1687 1 2 1 8020000 fffffe822d84f780 nmbd 1230 1 2 0 8020000 fffffe822d84fba0 openvpn 203 1 2 0 8020000 fffffe8230454340 openvpn 1861 1 2 0 8020000 fffffe8230454b80 openvpn 193 1 2 0 8020000 fffffe8228303320 smbd 500 1 2 0 8020000 fffffe8228303b60 saslauthd 1515 1 2 0 8020000 fffffe822f3cc300 saslauthd 2741 1 2 0 8020000 fffffe822f3cc720 saslauthd 3204 1 2 0 8020000 fffffe822ff71b20 saslauthd 3069 1 2 0 8020000 fffffe822f3ccb40 saslauthd 2758 1 2 0 8020000 fffffe82344c5180 winbindd 1785 1 2 0 8020000 fffffe822ff712e0 slpd 1458 1 2 0 8020000 fffffe822f5a56a0 winbindd 2908 1 2 0 8020000 fffffe822ff71700 smbd 2350 1 2 0 8020000 fffffe822f5a5280 upsmon 1735 1 2 0 8020000 fffffe822eb502c0 upsmon 2871 1 2 0 8020000 fffffe822eb50b00 upsd 1406 1 2 0 8020000 fffffe822eb506e0 upscode2 2736 1 2 0 8020000 fffffe82320ee220 winbindd 1755 1 2 1 8020000 fffffe822f60c2a0 winbindd 1743 1 2 0 8020000 fffffe822f60c6c0 cron 2394 1 2 0 8020000 fffffe822f60cae0 inetd 809 1 2 0 8020000 fffffe8233da1200 qmgr 810 1 2 0 8020000 fffffe822f5a5ac0 master 1912 1 2 0 8020000 fffffe8233da1620 sshd 1151 1 2 0 8020000 fffffe8233da1a40 timed 1265 1 2 1 8020000 fffffe823426e1e0 rwhod 1356 1 2 0 8020000 fffffe823426ea20 powerd 1669 1 2 0 8020000 fffffe823330b5c0 ntpd 1637 4 2 0 8020000 fffffe8234c1d1c0 nsd 1637 3 2 0 8020000 fffffe8234c1da00 nsd 1637 2 2 0 8020000 fffffe82344c59c0 nsd 1637 1 2 0 8020000 fffffe82344c55a0 nsd 98 1 2 0 8020000 fffffe82362e8580 postgres 1700 1 2 0 8020000 fffffe82370c10a0 postgres 1555 9 2 0 8020000 fffffe82362e89a0 archiveopteryx 1555 8 2 3 8020000 fffffe81d16c2560 archiveopteryx 1555 7 2 0 8020000 fffffe81e9e20440 archiveopteryx 1555 6 5 0 1a020000 fffffe8100482a40 archiveopteryx 1555 1 3 1 8020000 fffffe8236560140 archiveopteryx tstile 1667 10 2 3 8020000 fffffe81e3fa7160 archiveopteryx 1667 9 2 0 8020000 fffffe81b41ee6a0 archiveopteryx 1667 7 5 0 1a020000 fffffe81ead53ac0 archiveopteryx 1667 6 2 0 8020000 fffffe81c51c0ba0 archiveopteryx 1667 4 2 0 8020000 fffffe822c8c3460 archiveopteryx 1667 1 3 1 8020000 fffffe82381d4940 archiveopteryx tstile 941 1 2 0 8020000 fffffe8236560560 archiveopteryx 1496 1 2 0 8020000 fffffe8232541120 postgres 1530 1 2 1 8020000 fffffe8232541540 postgres 1565 1 2 0 8020000 fffffe8232541960 postgres 1558 1 3 2 8060000 fffffe82381d4520 postgres genput 1552 1 2 1 8020000 fffffe82381d4100 postgres 1446 4 2 0 8020000 fffffe8233f00500 dhcpd 1446 > 3 7 3 8020000 fffffe8233f00920 dhcpd 1446 2 2 0 8020000 fffffe82392444e0 dhcpd 1446 1 2 0 8020000 fffffe82392440c0 dhcpd 1196 1 2 3 8020000 fffffe823d348700 kdc 863 1 2 1 8020000 fffffe8239244900 rpc.lockd 1038 1 2 3 8020000 fffffe82370c14c0 rpc.statd 1007 17 2 0 8020000 fffffe82370c18e0 slave 1007 16 2 0 8020000 fffffe823973f080 slave 1007 15 2 0 8020000 fffffe823973f4a0 slave 1007 14 2 0 8020000 fffffe823973f8c0 slave 1007 13 2 0 8020000 fffffe8234680060 slave 1007 12 2 0 8020000 fffffe8234680480 slave 1007 11 2 0 8020000 fffffe82346808a0 slave 1007 10 2 0 8020000 fffffe82346c0040 slave 1007 9 2 0 8020000 fffffe82346c0460 slave 1007 8 2 0 8020000 fffffe82346c0880 slave 1007 7 2 0 8020000 fffffe82387fe020 slave 1007 6 2 0 8020000 fffffe82387fe440 slave 1007 5 2 0 8020000 fffffe82387fe860 slave 1007 4 2 0 8020000 fffffe8239cda000 slave 1007 3 2 0 8020000 fffffe823beed720 slave 1007 2 2 0 8020000 fffffe823beedb40 slave 1007 1 2 0 18020000 fffffe8239cda420 master 977 1 2 0 8020000 fffffe823c83c360 mountd 807 1 2 0 8020000 fffffe8239cda840 mount_mfs 734 1 2 3 8020000 fffffe823c83c780 rpcbind 699 7 2 0 8020000 fffffe823c83cba0 named 699 6 2 0 8020000 fffffe823d67f340 named 699 5 2 0 8020000 fffffe823d67f760 named 699 4 2 0 8020000 fffffe823d67fb80 named 699 3 2 0 8020000 fffffe823cead320 named 699 2 2 0 8020000 fffffe823d3482e0 named 699 1 2 0 8020000 fffffe823d3452c0 named 345 1 2 0 8020000 fffffe823d348b20 syslogd 1 1 2 0 8020000 fffffe81005ad640 init 0 71 3 0 200 fffffe823beed300 ipmi0 ipmi0 0 70 3 3 200 fffffe8100401600 physiod physiod 0 69 3 1 200 fffffe823d75b680 radeonfb radeonfb 0 68 3 3 200 fffffe823d75baa0 aiodoned aiodoned 0 67 3 2 200 fffffe81004011e0 ioflush genput 0 66 3 3 200 fffffe810054a660 pgdaemon pgdaemon 0 65 3 0 200 fffffe810054a240 ttm_swap ttm_swap 0 62 3 0 200 fffffe810047f5e0 pfpurge pftm 0 61 3 3 200 fffffe8100401a20 usb1 usbevt 0 60 2 3 200 fffffe810047f1c0 usb0 0 59 3 3 200 fffffe81003f91a0 atapibus1 sccomp 0 57 3 1 200 fffffe810047fa00 usb2 usbevt 0 56 3 0 200 fffffe810054aa80 usb3 usbevt 0 55 3 0 200 fffffe81005ad220 cryptoret crypto_w 0 54 3 0 200 fffffe81005ada60 unpgc unpgc 0 53 3 3 200 fffffe8100482200 atapibus0 sccomp 0 51 2 2 200 fffffe8100482620 vmem_rehash 0 42 3 0 200 fffffe81003f95c0 atabus3 atath 0 41 3 3 200 fffffe81003f99e0 atabus2 atath 0 40 3 1 200 fffffe810046d180 atabus1 atath 0 39 3 2 200 fffffe810046d5a0 atabus0 atath 0 38 3 0 200 fffffe810046d9c0 usbtask-dr usbtsk 0 37 3 0 200 fffffe8100169160 usbtask-hc usbtsk 0 36 2 1 200 fffffe8100169580 lnxsyswq 0 > 35 7 2 200 fffffe81001699a0 ipmi 0 34 3 3 200 fffffe81000ca140 xcall/3 xcall 0 33 1 3 200 fffffe81000ca560 softser/3 0 > 32 7 3 200 fffffe81000ca980 softclk/3 0 31 1 3 200 fffffe81000b1120 softbio/3 0 30 1 3 200 fffffe81000b1540 softnet/3 0 29 1 3 201 fffffe81000b1960 idle/3 0 28 3 2 200 fffffe8100087100 xcall/2 xcall 0 27 1 2 200 fffffe8100087520 softser/2 0 26 1 2 200 fffffe8100087940 softclk/2 0 25 1 2 200 fffffe81000850e0 softbio/2 0 24 1 2 200 fffffe8100085500 softnet/2 0 23 1 2 201 fffffe8100085920 idle/2 0 22 3 1 200 fffffe810007d0c0 xcall/1 xcall 0 21 1 1 200 fffffe810007d4e0 softser/1 0 20 3 1 200 fffffe810007d900 softclk/1 tstile 0 19 1 1 200 fffffe810007e0a0 softbio/1 0 18 1 1 200 fffffe810007e4c0 softnet/1 0 17 1 1 201 fffffe810007e8e0 idle/1 0 16 3 0 200 fffffe823db03080 sysmon smtaskq 0 15 3 1 200 fffffe823db034a0 pmfsuspend pmfsuspend 0 14 3 3 200 fffffe823db038c0 pmfevent pmfevent 0 13 3 0 200 fffffe823eb12060 sopendfree sopendfr 0 12 3 1 200 fffffe823eb12480 nfssilly nfssilly 0 11 2 2 200 fffffe823eb128a0 cachegc 0 10 2 2 200 fffffe823f71f040 vrele 0 9 3 3 200 fffffe823f71f460 vdrain vdrain 0 8 3 0 200 fffffe823f71f880 modunload mod_unld 0 7 3 0 200 fffffe823f724020 xcall/0 xcall 0 > 6 7 0 200 fffffe823f724440 softser/0 0 5 1 0 200 fffffe823f724860 softclk/0 0 4 1 0 200 fffffe823f72a000 softbio/0 0 3 1 0 200 fffffe823f72a420 softnet/0 0 2 1 0 201 fffffe823f72a840 idle/0 0 1 2 2 200 ffffffff810e87c0 swapper crash> ^D barsoom# -tih -- Popularity is the hallmark of mediocrity. --Niles Crane, "Frasier"
