On Wed, Mar 09, 2016 at 02:22:25PM +1000, David Gwynne wrote:
> hey marc,
>
> is this reproducable?
>
> if so, next time it happens can you collect ifconfig, systat mb, and vmstat
> -m output? im looking for IFF_OACTIVE in ifconfig, an ALIVE value that is
> different to the CWM in systat mb, or really high numbers against the mbuf
> pools in vmstat -m output.
re0: flags=18843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,MPSAFE> mtu 9000
lladdr 60:a4:4c:e8:14:65
priority: 0
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 192.168.43.1 netmask 0xffffff00 broadcast 192.168.43.255
10 users Load 0.24 0.30 0.56 Fri Mar 11 21:38:27 2016
IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
System 0 256 210 251
2048 128 350
9216 5 6
lo0
rl0
re0 9216 5 2 1024 5
enc0
ral0
pflog0
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 46300 46116 34830793 1280 6961
32 12684 87284 57045447 640 42024
64 7641 7335 97503021 320 1393
128 17895 6969 21536810 160 2515
256 21756 41540 18131788 80 825948
512 3964 3884 11184466 40 319516
1024 2270 2714 19101981 20 3718863
2048 2447 95 3450609 10 702901
4096 1112 846 837606 5 821017
8192 1391 2 1500355 5 55181
16384 1051 0 608098 5 0
32768 14 0 1099 5 0
65536 1 0 991955 5 0
131072 6 0 4350009 5 0
262144 1 0 1 5 0
524288 5 0 32 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, rtable, UFS mount, dirhash, ACPI, subproc, exec, VM swap,
UVM amap, UVM aobj, USB, USB device, temp, DRM
32 devbuf, pcb, rtable, ifaddr, UFS mount, sem, dirhash, ACPI, in_multi,
ether_multi, exec, UVM amap, UVM aobj, USB, USB device, temp, DRM
64 devbuf, rtable, ifaddr, vnodes, UFS mount, dirhash, ACPI, proc,
NFS daemon, in_multi, VM swap, UVM amap, UVM aobj, USB, USB device,
NDP, temp, DRM
128 devbuf, pcb, rtable, ifaddr, sysctl, UFS mount, sem, dirhash, ACPI,
NFS srvsock, in_multi, miscfs mount, pfkey data, inodedep, UVM amap,
UVM aobj, USB, ip6_options, temp, DRM
256 devbuf, rtable, ifaddr, ioctlops, iov, vnodes, shm, VM map, dirhash,
ACPI, MSDOSFS node, exec, UVM amap, UVM aobj, USB, USB device, temp,
DRM
512 devbuf, ioctlops, iov, NFS mount, UFS mount, dirhash, ACPI, file desc,
MSDOSFS mount, ttys, newblk, UVM amap, UVM aobj, temp, DRM
1024 devbuf, pcb, sysctl, ioctlops, iov, mount, UFS mount, shm, dirhash,
file desc, proc, ttys, exec, UVM amap, UVM aobj, crypto data, temp,
DRM
2048 devbuf, pcb, ioctlops, UFS mount, dirhash, ACPI, file desc, VM swap,
UVM amap, UVM aobj, temp, DRM
4096 devbuf, pcb, ifaddr, ioctlops, UFS mount, shm, ACPI, file desc, proc,
ttys, UVM amap, memdesc, temp, DRM
8192 devbuf, pcb, UFS mount, file desc, ttys, pagedep, UVM amap, USB, temp,
DRM
16384 devbuf, UFS mount, NFS daemon, MSDOSFS mount, indirdep, temp, DRM
32768 devbuf, UFS quota, UFS mount, ISOFS mount, inodedep, indirdep, temp,
DRM
65536 devbuf, temp, DRM
131072 devbuf, temp, DRM
262144 DRM
524288 devbuf, MSDOSFS fat, VM swap, DRM
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 3989 15395K 15432K157287K 1191022 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,524288
pcb 221 49K 60K157287K 59170 0 0
16,32,128,1024,2048,4096,8192
rtable 98 7K 11K157287K 25307 0 0 16,32,64,128,256
ifaddr 168 18K 18K157287K 176 0 0
32,64,128,256,4096
sysctl 2 2K 2K157287K 2 0 0 128,1024
ioctlops 0 0K 4K157287K 20396 0 0
256,512,1024,2048,4096
iov 0 0K 1K157287K 342 0 0 256,512,1024
mount 14 14K 15K157287K 30 0 0 1024
NFS mount 2 1K 1K157287K 2 0 0 512
vnodes 62 5K 81K157287K 15717 0 0 64,256
UFS quota 1 32K 32K157287K 1 0 0 32768
UFS mount 37 101K 101K157287K 37 0 0
16,32,64,128,512,1024,2048,4096,8192,16384,32768
shm 5 14K 22K157287K 581 0 0 256,1024,4096
VM map 2 1K 1K157287K 2 0 0 256
sem 2 1K 1K157287K 2 0 0 32,128
dirhash 798 149K 568K157287K 94077 0 0
16,32,64,128,256,512,1024,2048
ACPI 4268 494K 558K157287K 13552 0 0
16,32,64,128,256,512,2048,4096
file desc 20 55K 111K157287K 592 0 0
512,1024,2048,4096,8192
proc 20 11K 11K157287K 20 0 0 64,1024,4096
subproc 0 0K 1K157287K 1 0 0 16
NFS srvsock 1 1K 1K157287K 1 0 0 128
NFS daemon 5 17K 17K157287K 5 0 0 64,16384
in_multi 15 1K 1K157287K 17 0 0 32,64,128
ether_multi 2 1K 1K157287K 4 0 0 32
ISOFS mount 1 32K 32K157287K 1 0 0 32768
MSDOSFS mount 2 17K 18K157287K 7 0 0 512,16384
MSDOSFS fat 1 3728K 13044K157287K 6 0 0 524288
MSDOSFS node 339 85K 9188K157287K 291185 0 0 256
ttys 408 1724K 1724K157287K 408 0 0
512,1024,4096,8192
exec 0 0K 5K157287K 19595 0 0 16,32,256,1024
miscfs mount 2 1K 1K157287K 2 0 0 128
pfkey data 1 1K 1K157287K 2 0 0 128
pagedep 1 8K 8K157287K 1 0 0 8192
inodedep 1 32K 146K157287K 19064 0 0 128,32768
newblk 1 1K 1K157287K 1 0 0 512
indirdep 0 0K 320K157287K 235 0 0 16384,32768
VM swap 7 595K 595K157287K 7 0 0
16,64,2048,524288
UVM amap 88842 15234K 26186K157287K 17828333 0 0
16,32,64,128,256,512,1024,2048,4096,8192
UVM aobj 3002 336K 975K157287K 4307524 0 0
16,32,64,128,256,512,1024,2048
USB 94 26K 28K157287K 145 0 0
16,32,64,128,256,8192
USB device 17 2K 2K157287K 19 0 0 16,32,64,256
memdesc 1 4K 4K157287K 1 0 0 4096
crypto data 1 1K 1K157287K 1 0 0 1024
ip6_options 1 1K 1K157287K 8361 0 0 128
NDP 6 1K 1K157287K 6 0 0 64
temp 202 84K 3044K157287K 65651777 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072
DRM 15881 16176K 16241K157287K181526338 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144,524288
Memory Totals: In Use Free Requests
54440K 23409K 271074075
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
phpool 112 66304 0 4015 256 66 190 248 0 8 0
extentpl 40 109 0 44 1 0 1 1 0 8 0
pmappl 176 11603 0 92 45 39 6 6 0 8 0
pvpl 32 321560436 0 541892 16886 8876 8010 9420 0 265 0
pdppl 4096 11603 0 92 45 31 14 16 0 8 0
vmsppl 256 11603 0 92 31 23 8 8 0 8 0
vmmpepl 168 54872277 0 95612 154766 149685 5081 6807 0 357 0
vmmpekpl 168 25727463 0 203 25 15 10 12 0 8 0
uaddr 24 11604 0 93 1 0 1 1 0 8 0
uaddrbest 32 2 0 2 1 0 1 1 0 8 0
uaddrrnd 24 11604 0 93 1 0 1 1 0 8 0
aobjpl 64 3497700 0 3001 1131 1058 73 817 0 8 0
dma16 16 12 0 0 7 7 0 1 0 8 0
dma32 32 17 0 0 5 5 0 1 0 8 0
dma64 64 18 0 0 8 8 0 1 0 8 0
dma256 256 30 0 0 11 11 0 1 0 8 0
dma512 512 8 0 2 1 0 1 1 0 8 0
dma4096 4096 1 0 0 1 1 0 1 0 8 0
amappl 72 16468743 0 76401 6360 4532 1828 1983 0 75 0
anonpl 16 160997557 0 281042 5123 2372 2751 3027 0 507 0
bufpl 256 16294819 0 17712 6962 5447 1515 2423 0 8 0
mbufpl 256 1375472166 0 212 9310 9280 30 251 1 384 1
mtagpl 72 8 0 0 8 8 0 1 0 8 0
mcl2k 2048 377863140 0 128 12188 12167 21 350 1 768 5
mcl9k 9216 2526197 0 5 11 9 2 6 1 439 1
sockpl 392 282422 0 262 388 351 37 78 0 8 1
procpl 568 23107 0 253 60 36 24 33 0 8 0
processpl 672 11632 0 121 68 57 11 13 0 8 0
zombiepl 144 11522 0 11 2 1 1 1 0 8 0
ucredpl 96 7208 0 67 2 0 2 2 0 8 0
pgrppl 40 2595 0 57 1 0 1 1 0 8 0
sessionpl 64 263 0 36 1 0 1 1 0 8 0
lockfpl 88 251457 0 27 5 4 1 1 0 8 0
filepl 112 6128880 0 880 158 123 35 51 0 8 0
fdescpl 440 11604 0 93 76 65 11 15 0 8 0
pipepl 120 14518 0 93 11 7 4 5 0 8 0
kqueuepl 320 529 0 31 6 3 3 4 0 8 0
knotepl 112 9333051 0 176 228 220 8 27 0 8 0
sigapl 432 11603 0 92 61 50 11 15 0 8 0
scxspl 192 9394066 0 0 8677 8676 1 2 0 8 1
drmobjpl 704 2723527 0 2845 1230 963 267 274 0 8 2
ehcixfer 264 370 0 29 8 6 2 4 0 8 0
ohcixfer 224 75 0 4 1 0 1 1 0 8 0
nfsreqpl 96 8823202 0 0 346 345 1 1 0 8 1
namei 1024 46752276 0 0 6904 6903 1 2 0 8 1
vnodes 200 38750 0 38750 2040 0 2040 2040 0 8 0
uvmvnodes 72 38750 0 38750 705 0 705 705 0 8 0
rtmask 32 13 0 9 1 0 1 1 0 8 0
nchpl 144 23505607 0 5924 518 298 220 220 0 8 0
ffsino 240 21831343 0 37955 11965 9693 2272 2279 0 8 0
dino1pl 128 21831343 0 37955 5445 4199 1246 1250 0 8 0
pagedep 128 139147 0 1 4516 4515 1 202 0 8 0
inodedep 152 674976 0 2 13380 13379 1 1203 0 8 0
newblk 64 708420 0 0 15157 15157 0 1 0 8 0
bmsafemap 64 60166 0 0 7449 7449 0 7 0 8 0
allocdir 128 317701 0 0 8616 8616 0 253 0 8 0
indirdep 56 5356 0 0 544 544 0 3 0 8 0
allocindir 104 390719 0 0 8690 8690 0 154 0 8 0
freefrag 64 17146 0 0 6537 6537 0 3 0 8 0
freeblks 192 149688 0 0 10739 10739 0 353 0 8 0
freefile 48 297523 0 0 5567 5567 0 123 0 8 0
diradd 56 564041 0 1 5629 5628 1 443 0 8 0
mkdir 56 116870 0 0 1024 1024 0 119 0 8 0
dirrem 64 559201 0 1 6232 6231 1 228 0 8 0
newdirblk 32 495 0 0 16 16 0 1 0 8 0
dirhash 1024 204546 0 885 6541 6430 111 247 0 8 0
nfsnodepl 472 124903 0 259 4167 4134 33 4039 0 8 0
tmpfs_dirent 48 836172 0 91 1469 1467 2 1071 0 8 0
tmpfs_node 240 836158 0 182 7212 7193 19 5231 0 8 0
shmpl 112 18813 0 1 5 4 1 1 0 8 0
pfrule 1336 19 0 3 2 1 1 2 0 8 0
pfstate 312 414011 931 58 4799 4789 10 770 0 8 0
pfstkey 104 414011 0 58 1218 1212 6 264 0 8 0
pfstitem 24 414011 0 58 193 190 3 61 0 8 0
pfosfpen 112 1420 0 710 21 0 21 21 0 8 0
pfosfp 40 840 0 420 5 0 5 5 0 8 0
rttmr 72 2102 0 1 21 20 1 3 0 8 0
tcpcb 560 182261 0 34 424 417 7 41 0 8 0
tcpqe 32 104619 0 0 6869 6868 1 5 0 8 1
sackhl 24 15347 0 0 469 469 0 1 0 8 0
syncache 248 19223 0 0 2857 2856 1 2 0 8 1
rtentry 184 1882 0 34 13 10 3 9 0 8 0
plimitpl 152 354 0 36 2 0 2 2 0 8 0
inpcbpl 288 223271 0 54 329 321 8 41 0 8 1
arp 56 32 0 5 1 0 1 1 0 8 0
In use 76740K, total allocated 110224K; utilization 69.6%
