Userland is non-responsive, machine is pingable, tcp connections open
but no banner from ssh. No failed pool requests. This kernel is from
today's snapshot but I saw the same with one from a couple of days
ago. Is there anything else I can get that might be useful?
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
99554 86967 57409 55 3 0x2 vp cc
57409 23557 97377 55 3 0x82 wait cc
97377 51254 49407 55 3 0x10008a pause sh
71034 186155 65198 0 3 0x11 vp perl
49407 183801 58608 55 3 0x82 wait gmake
58608 251568 90720 55 3 0x10008a pause sh
90720 294385 26849 55 3 0x82 wait gmake
26849 434857 31480 55 3 0x100088 pause sh
31480 479316 1945 55 3 0x10008a pause sh
1945 53261 1392 55 3 0x82 wait gmake
1392 297593 28991 55 3 0x100088 pause sh
28991 101756 11650 55 3 0x10008a pause sh
11650 273060 70062 55 3 0x82 wait gmake
70062 380995 21324 55 3 0x82 wait gmake
21324 380494 20357 55 3 0x10008a pause make
20357 495141 79040 55 3 0x10008a pause sh
79040 411698 40069 55 3 0x10008a pause make
40069 407214 61289 55 3 0x10008a pause sh
61289 440156 65198 55 3 0x10008a pause make
16484 143829 63578 55 3 0x82 nanosleep perl
63578 247597 69857 55 3 0x10008a pause sh
69857 3708 28018 55 3 0x10008a pause make
28018 161747 1 55 3 0x10008a pause sh
78305 185109 40308 1000 3 0x100083 ttyin ksh
65198 454438 69872 0 3 0x93 wait perl
69872 91535 40308 1000 3 0x10008b pause ksh
40308 108204 1 1000 3 0x100080 kqread tmux
72632 510504 69073 1000 3 0x100083 kqread tmux
69073 166246 39096 1000 3 0x10008b pause ksh
39096 474432 39165 1000 3 0x10 vp sshd
39165 380864 95218 0 3 0x92 poll sshd
19837 75515 1 0 3 0x100003 vp getty
61 140725 1 0 3 0x100010 vp cron
33247 144573 1 110 3 0x100090 poll sndiod
85245 294054 1 99 3 0x100090 poll sndiod
20071 339430 77361 95 3 0x100092 kqread smtpd
31714 216717 77361 103 3 0x100092 kqread smtpd
38145 373966 77361 95 3 0x100092 kqread smtpd
73235 449750 77361 95 3 0x100092 kqread smtpd
52512 523411 77361 95 3 0x100092 kqread smtpd
25217 17706 77361 95 3 0x100092 kqread smtpd
77361 512649 1 0 3 0x100080 kqread smtpd
95218 352524 1 0 3 0x80 select sshd
28640 338771 0 0 3 0x14280 nfsidl nfsio
30707 131410 0 0 3 0x14280 nfsidl nfsio
26109 142203 0 0 3 0x14280 nfsidl nfsio
61054 453416 0 0 3 0x14280 nfsidl nfsio
20679 124381 1 0 3 0x80 poll rpc.statd
75142 494960 1 28 3 0x100090 poll portmap
13394 497677 1 0 3 0x100000 vp ntpd
56991 117256 27035 83 3 0x100092 poll ntpd
27035 498377 1 83 3 0x100092 poll ntpd
19071 360785 2016 74 3 0x100090 bpf pflogd
2016 326372 1 0 3 0x80 netio pflogd
75485 263260 29155 73 3 0x100090 kqread syslogd
29155 379800 1 0 3 0x100082 netio syslogd
9314 271265 1 77 3 0x100090 poll dhclient
77002 222287 1 0 3 0x80 poll dhclient
4332 479844 1 0 3 0x80 mfsidl mount_mfs
58334 330646 0 0 3 0x14200 pgzero zerothread
15557 331142 0 0 3 0x14200 aiodoned aiodoned
34557 432814 0 0 3 0x14200 syncer update
82663 208419 0 0 3 0x14200 cleaner cleaner
51853 347618 0 0 3 0x14200 reaper reaper
18753 499821 0 0 3 0x14200 pgdaemon pagedaemon
59831 415568 0 0 3 0x14200 bored crynlk
29354 478337 0 0 3 0x14200 bored crypto
33898 6377 0 0 3 0x14200 pftm pfpurge
33736 115197 0 0 3 0x14200 usbtsk usbtask
54044 43212 0 0 3 0x14200 usbatsk usbatsk
57344 273215 0 0 3 0x14200 bored softnet
88556 58351 0 0 3 0x14200 bored systqmp
69342 477649 0 0 3 0x14200 bored systq
25767 494373 0 0 3 0x40014200 bored softclock
*43785 51419 0 0 7 0x40014200 idle0
9892 420912 0 0 3 0x14200 kmalloc kmthread
1 54365 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb> sh re
x0 0x500
x1 0xffffff8022a8a000
x2 0
x3 0xffffffff
x4 0
x5 0xffffff8032b09000
x6 0xffffff801a4508f0
x7 0
x8 0x1
x9 0xffffff800058309c pluart_intr
x10 0xb
x11 0x60000005
x12 0
x13 0
x14 0x67
x15 0
x16 0x3ffffc3020
x17 0
x18 0xffffff8032b0dc40
x19 0xffffff8010374c00 _end+0xfa55d00
x20 0xffffff8010358470 _end+0xfa39570
x21 0xffffff8022a8a000
x22 0xffffff8010374c70 _end+0xfa55d70
x23 0xffffff8010374e08 _end+0xfa55f08
x24 0xffffff800080b004 hz
x25 0x3c
x26 0xffffff8000884b94 db_console
x27 0x3
x28 0xffffffffffffffff
x29 0xffffff8032b0dc80
x30 0xffffff8010370400 _end+0xfa51500
sp 0xffffff8032b0dc40
spsr 0xa00003c5
elr 0xffffff800046817c db_enter
lr 0xffffff8000583200 pluart_intr+0x164
pluart_intr+0x164:
ddb> sh bcst
Current Buffer Cache status:
numbufs 428744 busymapped 1, delwri 0
kvaslots 2047 avail kva slots 2046
bufpages 1620725, dmapages 1620725, dirtypages 0
pendingreads 24, pendingwrites 6
highflips 0, highflops 0, dmaflips 0
ddb> sh uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
2026672 VM pages: 74138 active, 99952 inactive, 0 wired, 103286 free (12916 z
ero)
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
pages 0 anon, 0 vnode, 0 vtext
freemin=67555, free-target=90073, inactive-target=0, wired-max=675557
faults=381390538, traps=0, intrs=0, ctxswitch=9483450 fpuswitch=0
softint=5031606, syscalls=234541519, kmapent=17
fault counts:
noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
ok relocks(total)=209237(209239), anget(retries)=156026897(0), amapcopy=998
93791
neighbor anon/obj pg=18888683/197482855, gets(lock/unlock)=155221275/209239
cases: anon=122743797, anoncow=33283100, obj=141166972, prcopy=14054301, pr
zero=70142365
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1, nanon=0, nanonneeded=0 nfreeanon=0
swpages=2162687, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0xffffff80008a53d0
ddb> sh all pools
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
arp 56 6 0 3 1 0 1 1 0 8 0
inpcbpl 288 184 0 162 3 1 2 3 0 8 0
plimitpl 152 34871 0 34839 48 46 2 2 0 8 0
rtentry 112 30 0 2 1 0 1 1 0 8 0
syncache 264 27 0 27 9 9 0 1 0 8 0
sackhl 24 174 0 174 118 118 0 1 0 8 0
tcpqe 32 24 0 24 6 6 0 1 0 8 0
tcpcb 560 80 0 70 4 3 1 3 0 8 0
pffrag 112 113035 0 113035 102 102 0 1 0 4 0
pffrent 40 675524 0 675524 102 102 0 1 0 8 0
pfosfp 40 846 0 423 5 0 5 5 0 8 0
pfosfpen 112 1428 0 714 21 0 21 21 0 8 0
pfstitem 24 3190 0 3174 1 0 1 1 0 8 0
pfstkey 112 3190 0 3174 1 0 1 1 0 8 0
pfstate 312 3190 0 3174 34 31 3 3 0 8 0
pfrule 1360 19 0 16 2 1 1 2 0 8 0
art_heap8 4096 1 0 0 1 0 1 1 0 8 0
art_heap4 256 99 0 0 7 0 7 7 0 8 0
art_table 32 100 0 0 1 0 1 1 0 8 0
art_node 16 30 0 3 1 0 1 1 0 8 0
shmpl 112 1 0 1 1 1 0 1 0 8 0
nfsnodepl 472 23763 0 0 2971 0 2971 2971 0 8 0
dirhash 1024 5345 0 3552 283 58 225 236 0 8 0
newdirblk 32 1845 0 1845 49 49 0 2 0 8 0
dirrem 64 103140 0 103140 658 658 0 92 0 8 0
mkdir 56 52732 0 52732 503 503 0 82 0 8 0
diradd 56 415732 0 415732 3331 3331 0 781 0 8 0
freefile 48 75692 0 75692 404 404 0 66 0 8 0
freeblks 192 65111 0 65111 1534 1534 0 241 0 8 0
freefrag 64 58888 0 58888 322 322 0 13 0 8 0
allocindir 104 113054 0 113054 2028 2028 0 159 0 8 0
indirdep 56 6949 0 6949 51 51 0 4 0 8 0
allocdir 128 660165 0 660165 9004 9004 0 1423 0 8 0
bmsafemap 64 8034 0 8034 39 39 0 18 0 8 0
newblk 64 773219 0 773219 53 53 0 1 0 8 0
inodedep 152 424946 0 424946 10105 10105 0 2132 0 8 0
pagedep 128 31061 0 31061 617 617 0 99 0 8 0
dino1pl 128 509208 0 122122 12943 456 12487 12487 0 8 0
ffsino 240 509208 0 122122 23834 1064 22770 22770 0 8 0
nchpl 144 3002264 0 2999602 108 9 99 99 0 8 0
uvmvnodes 72 432244 0 0 7859 0 7859 7859 0 8 0
vnodes 200 432244 0 0 22750 0 22750 22750 0 8 0
namei 1024 19646319 0 19646319 14 13 1 1 0 8 1
nfsreqpl 96 233444 0 233444 130 130 0 1 0 8 0
xhcixfer 240 21 0 20 1 0 1 1 0 8 0
extentpl 40 21 0 8 1 0 1 1 0 8 0
scxspl 192 837067 0 837067 364 364 0 2 0 8 0
sigapl 432 528069 0 528011 168 161 7 11 0 8 0
knotepl 112 50981 0 50919 4 2 2 3 0 8 0
kqueuepl 320 26 0 16 3 2 1 2 0 8 0
pipepl 120 325488 0 325485 5 4 1 2 0 8 0
fdescpl 440 528070 0 528011 178 171 7 11 0 8 0
filepl 112 6305007 0 6304847 51 46 5 8 0 8 0
lockfpl 88 1764 0 1763 1 0 1 1 0 8 0
sessionpl 112 612 0 588 3 2 1 3 0 8 0
pgrppl 40 876 0 850 1 0 1 1 0 8 0
ucredpl 96 1836765 0 1836605 5 1 4 4 0 8 0
zombiepl 144 528013 0 528013 71 71 0 3 0 8 0
processpl 744 528092 0 528013 124 116 8 12 0 8 0
procpl 560 528197 0 528118 91 85 6 10 0 8 0
sockpl 392 526 0 443 30 21 9 13 0 8 0
mcl64k 65536 13458 0 13458 24 24 0 1 0 8 0
mcl16k 16384 1792 0 1792 42 42 0 1 0 8 0
mcl12k 12288 1926 0 1926 45 45 0 1 0 8 0
mcl9k 9216 1831747 0 1831731 273 271 2 7 0 8 0
mcl8k 8192 427217 0 427217 328 328 0 8 0 8 0
mcl4k 4096 632729 0 632729 302 302 0 10 0 8 0
mcl2k2 2112 1840 0 1840 85 85 0 1 0 8 0
mcl2k 2048 151075 0 151075 111 111 0 3 0 8 0
mtagpl 80 1 0 1 1 1 0 1 0 8 0
mbufpl 256 7662391 0 7662359 349 346 3 11 0 8 0
bufpl 256 1019674 0 590930 28742 1872 26870 27016 0 8 0
anonpl 16 117479766 0 117413683 2989 2699 290 459 0 509 0
amapchunkpl 152 5319107 0 5318222 1210 1175 35 72 0 158 0
amappl16 192 1879478 0 1877838 510 423 87 290 0 8 0
amappl15 184 113460 0 113447 18 17 1 2 0 8 0
amappl14 176 136516 0 136495 32 30 2 9 0 8 0
amappl13 168 74270 0 74263 9 8 1 2 0 8 0
amappl12 160 136145 0 136135 52 51 1 7 0 8 0
amappl11 152 104633 0 104564 35 31 4 9 0 8 0
amappl10 144 350501 0 350479 18 16 2 4 0 8 0
amappl9 136 321256 0 321236 126 124 2 54 0 8 0
amappl8 128 421800 0 421634 494 488 6 14 0 8 0
amappl7 120 130977 0 130958 76 75 1 4 0 8 0
amappl6 112 179474 0 179449 84 83 1 4 0 8 0
amappl5 104 197814 0 197754 175 173 2 28 0 8 0
amappl4 96 1913630 0 1913090 370 356 14 22 0 8 0
amappl3 88 921832 0 921568 407 398 9 31 0 8 0
amappl2 80 10417315 0 10408563 9206 9019 187 275 0 8 0
amappl1 72 63049893 0 63030370 39302 38926 376 643 0 8 0
amappl 72 2124125 0 2123721 92 83 9 14 0 75 0
dma4096 4096 1 0 1 1 1 0 1 0 8 0
dma512 512 3 0 2 1 0 1 1 0 8 0
dma256 256 7 0 7 1 1 0 1 0 8 0
dma64 64 2 0 2 1 1 0 1 0 8 0
dma32 32 6 0 6 1 1 0 1 0 8 0
dma16 16 2 0 2 1 1 0 1 0 8 0
vp 8192 64728981 0 64716405 152138 150566 1572 2427 3 8 0
pted 40 596975514 0 596903167 14657 13909 748 1274 1 8 1
pmap 48 528069 0 528011 2 0 2 2 1 8 1
aobjpl 64 2 0 1 1 0 1 1 0 8 0
uaddrrnd 24 528068 0 528011 1 0 1 1 0 8 0
uaddrbest 32 2 0 0 1 0 1 1 0 8 0
uaddr 24 528068 0 528011 1 0 1 1 0 8 0
vmmpekpl 168 1825149 0 1825068 36 27 9 20 0 8 0
vmmpepl 168 159416614 0 159375452 203585 201795 1790 3462 0 357 0
vmsppl 256 528069 0 528011 63 59 4 6 0 8 0
phpool 112 206405 0 154936 1784 313 1471 1471 0 8 0
