> Interesting. Could you enter ddb and report a trace of the ssh client
> stuck on "pmrwait"? At the same time, could you get "ps", "show uvm",
> "show all pools" and "show bcstats"?
I can't give you a trace (might be an md ddb limit though):
ddb> tr /t 0t373837
sh trace requires a trap frame... giving up
but here are the other data:
ddb> ps
PID TID PPID UID S FLAGS WAIT COMMAND
34039 45023 972 1500 3 0x100083 ttyin ksh
11596 373837 76176 1500 3 0x100003 pmrwait ssh
5573 153130 35412 1500 3 0x100083 piperd tee
76176 403395 35412 1500 3 0x100083 pipewr cvs
44420 331742 1 0 3 0x100003 biowait getty
28507 306701 1 0 3 0x100098 kqread cron
35412 213282 972 1500 3 0x10008b sigsusp ksh
972 692 33838 1500 3 0x98 kqread sshd-session
20768 12155 1 0 3 0x100090 kqread inetd
33838 127628 32996 0 3 0x92 kqread sshd-session
71683 242926 86447 95 3 0x1100092 kqread smtpd
19193 440880 86447 103 3 0x1100092 kqread smtpd
2773 352775 86447 95 3 0x1100092 kqread smtpd
80380 84874 86447 95 3 0x100092 kqread smtpd
78651 105250 86447 95 3 0x1100092 kqread smtpd
1019 424871 86447 95 3 0x1100092 kqread smtpd
86447 438433 1 0 3 0x100080 kqread smtpd
32996 45786 1 0 3 0x88 kqread sshd
36490 298831 0 0 3 0x14280 nfsidl nfsio
6332 393031 0 0 3 0x14280 nfsidl nfsio
68122 444454 0 0 3 0x14280 nfsidl nfsio
22508 377655 0 0 3 0x14280 nfsidl nfsio
74940 391699 1 0 3 0x80 kqread ypbind
50740 297093 1 28 3 0x1100090 kqread portmap
65418 281762 1 0 3 0x100080 kqread ntpd
36270 219733 96577 83 3 0x100092 kqread ntpd
96577 59236 1 83 3 0x1100092 kqread ntpd
41404 508876 16737 74 3 0x1100092 bpf pflogd
16737 10483 1 0 3 0x80 sbwait pflogd
16684 264180 50564 73 3 0x1100090 kqread syslogd
50564 259081 1 0 3 0x100082 sbwait syslogd
89773 212742 43428 115 3 0x100092 kqread slaacd
66261 152498 43428 115 3 0x100092 kqread slaacd
43428 518633 1 0 3 0x100080 kqread slaacd
45703 91458 0 0 3 0x14200 bored smr
41113 473956 0 0 3 0x14200 pgzero zerothread
30178 153074 0 0 3 0x14200 aiodoned aiodoned
11814 276588 0 0 3 0x14200 syncer update
12223 349500 0 0 3 0x14200 cleaner cleaner
73245 38967 0 0 3 0x14200 reaper reaper
6958 501461 0 0 3 0x14200 pgdaemon pagedaemon
50466 448966 0 0 3 0x14200 usbtsk usbtask
92161 46152 0 0 3 0x14200 usbatsk usbatsk
85123 485245 0 0 3 0x14200 bored softnet0
21723 410855 0 0 3 0x14200 bored systqmp
31670 472417 0 0 3 0x14200 bored systq
56424 407036 0 0 3 0x40014200 tmoslp softclock
* 8970 665 0 0 7 0x40014200 idle0
1 381577 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb> show uvm
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
14869 VM pages: 240 active, 0 inactive, 1 wired, 9939 free (1276 zero)
freemin=495, free-target=660, inactive-target=661, wired-max=4956
faults=87243805, traps=44051216, intrs=25502210, ctxswitch=13627270 fpuswitch
=0
softint=11413539, syscalls=44051214, kmapent=10
fault counts:
noram=736888, noanon=0, noamap=0, pgwait=7922, pgrele=0
relocks=2070902(29829), upgrades=0(0) anget(retries)=28667830(824941), amap
copy=11284265
neighbor anon/obj pg=16084760/50520499, gets(lock/unlock)=15508304/1250980
cases: anon=25011559, anoncow=3630690, obj=13784081, prcopy=1719204, przero
=43069761
daemon and swap counts:
woke=1997106, revs=100785872, scans=3839272, obscans=820131, anscans=237496
4
busy=0, freed=2029723, reactivate=644065, deactivate=9186210
pageouts=247720, pending=247720, nswget=842534
nswapdev=1
swpages=4194415, swpginuse=4654, swpgonly=4530 paging=0
kernel pointers:
objs(kern)=0x8c40b1ac
ddb> show all pools
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
rtpcb 76 19 0 17 1 0 1 1 0 8 0
rtentry 96 50 0 23 1 0 1 1 0 8 0
unpcb 92 84 0 27 2 0 2 2 0 8 0
syncache 256 1 0 1 1 0 1 1 0 8 1
tcpcb 532 12 0 0 1 0 1 1 0 8 0
arp 64 26 0 23 1 0 1 1 0 8 0
inpcb 232 4019 0 3993 2 0 2 2 0 8 0
nd6 76 1 0 0 1 0 1 1 0 8 0
pffrag 132 14 0 14 1 0 1 1 0 9 1
pffrnode 64 14 0 14 1 0 1 1 0 8 1
pffrent 24 63 0 63 1 0 1 1 0 8 1
pfosfp 28 1428 0 1005 3 0 3 3 0 8 0
pfosfpen 108 1428 0 714 20 0 20 20 0 8 0
pfstitem 12 74571 0 74546 1 0 1 1 0 8 0
pfstkey 128 74571 0 74546 3 0 3 3 0 8 1
pfstate 256 74571 0 74546 6 0 6 6 0 8 4
pfrule 1204 21 0 16 2 0 2 2 0 8 1
art_heap8 2048 1 0 0 1 0 1 1 0 8 0
art_heap4 128 145 0 23 4 0 4 4 0 8 0
art_table 32 146 0 23 1 0 1 1 0 8 0
art_node 24 50 0 23 1 0 1 1 0 8 0
semapl 76 2 0 2 1 0 1 1 0 8 1
nfsnodepl 388 41 0 35 2 0 2 2 0 8 0
dirhash 1024 4164 0 4109 68 56 12 29 0 8 3
dino2pl 256 554438 0 551798 259 93 166 166 0 8 0
dino1pl 128 3 0 3 1 0 1 1 0 8 1
ffsino 196 554441 0 551801 204 71 133 133 0 8 0
nchpl 88 1012727 0 1010067 61 1 60 60 0 8 0
uvmvnodes 52 2662 0 0 35 0 35 35 0 8 0
vnodes 128 2662 0 0 84 0 84 84 0 8 0
namei 1024 10035689 0 10035689 1 0 1 1 0 8 1
nfsreqpl 60 297 0 297 1 0 1 1 0 8 1
ohcixfer 156 38 0 36 1 0 1 1 0 8 0
ehcixfer 180 24 0 23 1 0 1 1 0 8 0
wdcxfer 116 2640764 0 2640763 1 0 1 1 0 8 0
kstatmem 156 9 0 0 1 0 1 1 0 8 0
extentpl 20 14 0 5 1 0 1 1 0 8 0
plimitpl 152 555 0 537 1 0 1 1 0 8 0
sigapl 292 207902 0 207853 6 0 6 6 0 8 2
knotepl 76 1196692 0 1196591 3 0 3 3 0 8 0
kqueuepl 104 567 0 537 1 0 1 1 0 8 0
pipepl 204 50765 0 50762 1 0 1 1 0 8 0
fdescpl 324 207885 0 207853 6 0 6 6 0 8 3
filepl 96 4800595 0 4800450 5 0 5 5 0 8 1
lockfpl 64 7415 0 7413 1 0 1 1 0 8 0
lockfspl 24 3568 0 3566 1 0 1 1 0 8 0
sessionpl 100 148 0 131 1 0 1 1 0 8 0
pgrppl 28 286 0 268 1 0 1 1 0 8 0
ucredpl 100 540 0 509 1 0 1 1 0 8 0
zombiepl 80 207853 0 207853 1 0 1 1 0 8 1
processpl 720 207902 0 207853 8 0 8 8 0 8 3
procpl 472 207902 0 207853 10 0 10 10 0 8 3
sockpl 324 4122 0 4037 8 0 8 8 0 8 0
mcl9k 9216 1 0 1 1 0 1 1 0 8 1
mcl8k 8192 4 0 4 1 0 1 1 0 8 1
mcl4k 4096 7 0 7 1 0 1 1 0 8 1
mcl2k 2048 98787 0 98782 3 0 3 3 0 8 2
mbufpl 256 226881 0 226864 3 0 3 3 0 8 1
bufpl 168 1978493 0 1978298 339 321 18 49 0 8 6
anonpl 16 49154086 0 49149438 2393 2345 48 150 0 4 4
amapchunkpl 80 5161631 0 5161296 55 0 55 55 0 82 44
amappl16 112 68127 0 68087 15 5 10 15 0 8 8
amappl15 108 31760 0 31760 1 0 1 1 0 8 1
amappl14 104 8734 0 8732 1 0 1 1 0 8 0
amappl13 100 28859 0 28859 1 0 1 1 0 8 1
amappl12 96 141987 0 141985 1 0 1 1 0 8 0
amappl11 92 19896 0 19895 1 0 1 1 0 8 0
amappl10 88 156320 0 156253 3 0 3 3 0 8 1
amappl9 84 23893 0 23889 2 0 2 2 0 8 1
amappl8 80 46897 0 46892 3 0 3 3 0 8 2
amappl7 76 32695 0 32687 2 0 2 2 0 8 1
amappl6 72 361599 0 361565 2 0 2 2 0 8 0
amappl5 68 215860 0 215807 3 0 3 3 0 8 1
amappl4 64 867582 0 867517 54 43 11 18 0 8 8
amappl3 60 196807 0 196799 5 0 5 5 0 8 3
amappl2 56 1290347 0 1290138 14 1 13 14 0 8 8
amappl1 52 4712450 0 4711404 20 0 20 20 0 8 4
amappl 56 1331007 0 1330884 10 0 10 10 0 57 6
dma512 512 4 0 4 1 0 1 1 0 8 1
pvpl 16 147950747 0 147950486 7550 7541 9 55 1 8 7
pmappl 20 207884 0 207853 1 0 1 1 0 8 0
aobjpl 44 2 0 0 1 0 1 1 0 8 0
uaoeltpl 84 591 0 588 1 0 1 1 0 8 0
uaddrrnd 12 207885 0 207853 1 0 1 1 0 8 0
uaddrbest 16 2 0 0 1 0 1 1 0 8 0
uaddr 12 207885 0 207853 1 0 1 1 0 8 0
vmmpekpl 100 5206384 0 5206366 1 0 1 1 0 8 0
vmmpepl 100 29627512 0 29624456 200 0 200 200 0 205 114
vmsppl 212 207884 0 207853 4 0 4 4 0 8 2
rwobjpl 28 6562024 0 6557842 50 11 39 46 0 8 8
phpool 60 449 0 149 5 0 5 5 0 8 0
ddb> show bcstats
Current Buffer Cache status:
numbufs 176 busymapped 0, delwri 3
kvaslots 185 avail kva slots 185
bufpages 692, dmapages 692, dirtypages 12
pendingreads 0, pendingwrites 4
highflips 0, highflops 0, dmaflips 0
