On 6/25/22 13:45, Stuart Henderson wrote:
On 2022-06-24, Boyd Stephens <bsteph...@netelysis.com> wrote:
On 6/23/22 05:34, Stuart Henderson wrote:
How do the following look?
pfctl -si
systat -b mbuf
vmstat -m
Comparing normal + failed might be useful too.
Are you using queues in pf?
The ifconfig output you included looks normal. (rxpause/txpause is
"has negotiated flow control" and doesn't indicate what flow control
is actually blocking)
Mr Stuart:
Again we are not using any queues in pf and I have included the
requested data below.
Over this past weekend we were able to anecdotally determine what
triggers the condition and error. The IT Director and his team powered
down the telco vendor's/upstream equipment into which the ix0 device is
connected. Once this was done the network gateway device became
inoperable and began again with the
"No buffer space available" as in
0# ping www.google.com
ping: Warning: www.google.com has multiple addresses; using 108.177.122.103
PING www.google.com (108.177.122.103): 56 data bytes
ping: sendmsg: No buffer space available
ping: wrote www.google.com 64 chars, ret=-1
ping: sendmsg: No buffer space available
ping: wrote www.google.com 64 chars, ret=-1
ping: sendmsg: No buffer space available
ping: wrote www.google.com 64 chars, ret=-1
and again only a reboot "quickly" resolved the issue.
While talking with the IT Director he shared that the last time that
these symptoms exhibited themselves was when this same upstream
equipment lost electrical power and the network gateway and its ix0
interface was attempting to transmit data into it.
Again the requested data is listed below and thank you and others for
taking the time to assist us with this issue.
Oh, here is a response from netstat -i for the ix0 interface under both
conditions
Name Mtu Network Address Ipkts Ifail Opkts
Ofail Colls
ix0 1500 198.64.189. 198.64.189.200 35222341 0 19088719
0 0
(Successfully operating)
ix0 1500 198.64.189. 198.64.189.200 89985788 0 51396747
155461 0
(Under Failure Conditions)
--
Boyd Stephens
(Successfully working)
# pfctl -si
Status: Enabled for 1 days 23:30:23 Debug: err
Interface Stats for none IPv4 IPv6
Bytes In 0 0
Bytes Out 0 0
Packets In
Passed 0 0
Blocked 0 0
Packets Out
Passed 0 0
Blocked 0 0
State Table Total Rate
current entries 3921
half-open tcp 1
searches 232641576 1360.3/s
inserts 1171172 6.8/s
removals 1170453 6.8/s
Counters
match 119170659 696.8/s
bad-offset 0 0.0/s
fragment 13 0.0/s
short 7 0.0/s
normalize 12 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 5579 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
translate 0 0.0/s
no-route 0 0.0/s
(Under failure conditions)
# pfctl -si
Status: Enabled for 2 days 05:18:56 Debug: err
Interface Stats for none IPv4 IPv6
Bytes In 0 0
Bytes Out 0 0
Packets In
Passed 0 0
Blocked 0 0
Packets Out
Passed 0 0
Blocked 0 0
State Table Total Rate
current entries 5626
half-open tcp 142
searches 272992115 1422.3/s
inserts 1456315 7.6/s
removals 1453891 7.6/s
Counters
match 139956626 729.2/s
bad-offset 0 0.0/s
fragment 17 0.0/s
short 7 0.0/s
normalize 14 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 6326 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
translate 0 0.0/s
no-route 0 0.0/s
(Successfully working)
# systat -b mbuf
1 users Load 0.06 0.03 0.00 mgmpha-corp-gw-fw00
07:29:36
IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
System mbufs 0 256 1158 147
mcl2k 2048 14 15
mcl2k2 2112 952 143
mcl4k 4096 0 7
mcl8k 8192 0 5
mcl9k 9216 0 2
mcl12k 12288 0 2
mcl16k 16384 0 2
mcl64k 65536 0 1
lo0
ix0 0 2048 117 10 255 117
1 2048 127 10 255 127
2 2048 135 10 255 135
3 2048 126 10 255 126
4 2048 135 10 255 135
5 2048 135 10 255 135
ix1
em0 2050 151 10 256 151
em1 2050 26 10 256 26
enc0
pflog0
(Under Failure Conditions)
# systat -b mbuf
1 users Load 0.09 0.05 0.01 mgmpha-corp-gw-fw00
13:18:13
IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
System mbufs 0 256 2692 178
mcl2k 2048 16 15
mcl2k2 2112 2473 173
mcl4k 4096 0 7
mcl8k 8192 0 5
mcl9k 9216 0 2
mcl12k 12288 0 2
mcl16k 16384 0 2
mcl64k 65536 0 1
lo0
ix0 0 2048 117 10 255 117
1 2048 128 10 255 128
2 2048 136 10 255 136
3 2048 126 10 255 126
4 2048 135 10 255 135
5 2048 135 10 255 135
ix1
em0 2050 153 10 256 153
em1 2050 26 10 256 26
enc0
pflog0
(Successfully Working)
# vmstat -m
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 1238 554 13363 1280 0
32 2374 698 163909 640 29
64 2935 73 30055 320 0
128 21771 21 641958 160 163
256 388 28 10303 80 0
512 116 4 4872 40 0
1024 101 111 22197 20 15071
2048 1598 12 5119 10 1304
4096 1065 2 11957 5 0
8192 59 2 443 5 0
16384 9 0 439 5 0
32768 8 0 47 5 0
65536 1 0 5661 5 0
131072 2 0 2 5 0
262144 1 0 1 5 0
524288 3 0 3 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, rtable, UFS mount, dirhash, ACPI, proc, exec, VM
swap,
UVM amap, UVM aobj, USB, USB device, temp
32 devbuf, pcb, rtable, ifaddr, sysctl, UFS mount, sem, dirhash,
ACPI,
in_multi, ether_multi, exec, UVM amap, USB, USB device, NDP, temp
64 devbuf, rtable, ifaddr, counters, vnodes, dirhash, ACPI, proc,
in_multi, VM swap, UVM amap, USB, USB device, temp
128 devbuf, rtable, ifaddr, sysctl, counters, vnodes, UFS mount, sem,
dirhash, ACPI, NFS srvsock, in_multi, UVM amap, USB, USB
device, temp
256 devbuf, rtable, ifaddr, counters, ioctlops, iov, vnodes, UFS
mount,
shm, VM map, dirhash, ACPI, exec, tdb, UVM amap, USB, USB device,
temp
512 devbuf, counters, ioctlops, UFS mount, dirhash, ACPI, file
desc, ttys,
newblk, USB HC, temp, kqueue
1024 devbuf, pcb, sysctl, counters, ioctlops, mount, shm, ACPI,
proc, ttys,
exec, UVM amap, USB device, crypto data, temp
2048 devbuf, ioctlops, iov, UFS mount, ACPI, VM swap, UVM aobj, USB,
USB device, temp, kqueue
4096 devbuf, ifaddr, counters, ioctlops, UFS mount, proc, USB, USB
device,
memdesc, temp
8192 devbuf, counters, proc, ttys, pagedep, UVM amap, temp, SYN cache
16384 devbuf, NFS daemon, MSDOSFS mount, ttys, temp
32768 devbuf, UFS quota, UFS mount, ISOFS mount, inodedep
65536 devbuf, temp
131072 devbuf
262144 devbuf
524288 VM swap, temp
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Size(s)
devbuf 6187 8602K 8603K 78644K 7989 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144
pcb 13 9K 9K 78644K 13 0 16,32,1024
rtable 225 9K 10K 78644K 4505 0 16,32,64,128,256
ifaddr 59 11K 12K 78644K 76 0
32,64,128,256,4096
sysctl 3 2K 2K 78644K 3 0 32,128,1024
counters 124 102K 102K 78644K 124 0
64,128,256,512,1024,4096,8192
ioctlops 0 0K 4K 78644K 2252 0
256,512,1024,2048,4096
iov 0 0K 2K 78644K 27 0 256,2048
mount 5 5K 5K 78644K 5 0 1024
vnodes 1249 79K 79K 78644K 1293 0 64,128,256
UFS quota 1 32K 32K 78644K 1 0 32768
UFS mount 21 59K 59K 78644K 21 0
16,32,128,256,512,2048,4096,32768
shm 2 2K 2K 78644K 2 0 256,1024
VM map 2 1K 1K 78644K 2 0 256
sem 2 1K 1K 78644K 2 0 32,128
dirhash 126 24K 24K 78644K 153 0
16,32,64,128,256,512
ACPI 22718 2526K 2553K 78644K 588063 0
16,32,64,128,256,512,1024,2048
file desc 1 1K 1K 78644K 1 0 512
proc 117 98K 114K 78644K 1488 0
16,64,1024,4096,8192
NFS srvsock 1 1K 1K 78644K 1 0 128
NFS daemon 1 16K 16K 78644K 1 0 16384
in_multi 13 1K 1K 78644K 13 0 32,64,128
ether_multi 3 1K 1K 78644K 3 0 32
ISOFS mount 1 32K 32K 78644K 1 0 32768
MSDOSFS mount 1 16K 16K 78644K 1 0 16384
ttys 31 149K 149K 78644K 31 0
512,1024,8192,16384
exec 0 0K 3K 78644K 8273 0 16,32,256,1024
tdb 3 1K 1K 78644K 3 0 256
pagedep 1 8K 8K 78644K 1 0 8192
inodedep 1 32K 32K 78644K 1 0 32768
newblk 1 1K 1K 78644K 1 0 512
VM swap 7 587K 587K 78644K 7 0 16,64,2048,524288
UVM amap 603 70K 72K 78644K 41861 0
16,32,64,128,256,1024,8192
UVM aobj 3 3K 3K 78644K 3 0 16,2048
USB 39 30K 30K 78644K 46 0
16,32,64,128,256,2048,4096
USB device 16 13K 13K 78644K 16 0
16,32,64,128,256,1024,2048,4096
USB HC 1 1K 1K 78644K 1 0 512
memdesc 1 4K 4K 78644K 1 0 4096
crypto data 1 1K 1K 78644K 1 0 1024
NDP 7 1K 1K 78644K 7 0 32
temp 30 4680K 4748K 78644K 251987 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,65536,524288
kqueue 48 80K 101K 78644K 2048 0 512,2048
SYN cache 2 16K 16K 78644K 2 0 8192
Memory Totals: In Use Free Requests
17291K 206K 910330
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg
Maxpg Idle
phpool 112 19546 0 5902 179 9 170 171 0
8 0
extentpl 40 113 0 47 1 0 1 1 0
8 0
pmappl 216 7542 0 39 3 0 3 3 0
8 0
pvpl 32 7396179 0 19096 1959 1804 155 1444 0
265 1
pdppl 4096 15093 0 87 6440 6353 87 101 0
8 0
rwobjpl 24 624121 0 23219 170 30 140 141 0
8 0
vmsppl 272 7542 0 39 4 1 3 4 0
8 0
vmmpepl 168 1767347 0 4072 1652 1474 178 364 0
357 0
vmmpekpl 168 1135131 0 31 2 0 2 2 0
8 0
uaddr 24 7543 0 40 1 0 1 1 0
8 0
uaddrbest 32 2 0 2 1 0 1 1 0
8 0
uaddrrnd 24 7543 0 40 1 0 1 1 0
8 0
aobjpl 72 2 0 2 1 0 1 1 0
8 0
dma16 16 29 0 0 1 1 0 1 0
8 0
dma32 32 9 0 0 1 1 0 1 0
8 0
dma64 64 10 0 0 1 1 0 1 0
8 0
dma128 128 1 0 0 1 1 0 1 0
8 0
dma256 256 10 0 0 1 1 0 1 0
8 0
dma512 512 7 0 1 1 0 1 1 0
8 0
dma4096 4096 1 0 0 1 1 0 1 0
8 0
amappl 88 31581 0 368 30 21 9 23 0
92 0
amappl1 80 650872 0 1393 117 88 29 42 0
8 0
amappl2 88 32256 0 258 50 42 8 30 0
8 0
amappl3 96 12941 0 84 9 6 3 9 0
8 0
amappl4 104 39025 0 67 29 26 3 24 0
8 0
amappl5 112 6466 0 35 16 14 2 3 0
8 0
amappl6 120 10873 0 55 5 2 3 4 0
8 0
amappl7 128 976 0 13 2 1 1 2 0
8 0
amappl8 136 5778 0 20 13 12 1 13 0
8 0
amappl9 144 146 0 4 1 0 1 1 0
8 0
amappl10 152 43 0 2 1 0 1 1 0
8 0
amappl11 160 944 0 46 8 6 2 3 0
8 0
amappl12 168 188 0 0 9 9 0 3 0
8 0
amappl13 176 207 0 7 13 12 1 1 0
8 0
amappl14 184 54 0 0 6 6 0 1 0
8 0
amappl15 192 4731 0 18 3 2 1 2 0
8 0
amappl16 200 1170 0 12 19 18 1 17 0
8 0
amapchunkpl 152 80921 0 945 186 149 37 114 0
158 0
anonpl 24 1479786 0 9307 382 325 57 268 0
1519 0
bufpl 288 58712 0 47209 3375 2 3373 3373 0
8 0
mbufpl 256 127852494 0 1174 564 471 93 147 0
8 0
mcl2k 2048 47667 0 22 20 10 10 15 0
8 0
mcl2k2 2112 124875135 0 952 611 526 85 143 0
8 0
mcl4k 4096 2740 0 1 36 32 4 7 0
8 0
mcl8k 8192 522 0 0 5 0 5 5 0
8 0
mcl9k 9216 64 0 0 2 0 2 2 0
8 0
mcl12k 12288 41 0 0 2 0 2 2 0
8 0
mcl16k 16384 73 0 0 2 0 2 2 0
8 0
mcl64k 65536 8 0 0 1 0 1 1 0
8 0
sockpl 448 30040 0 93 34 23 11 13 0
8 0
procpl 672 7612 0 63 6 0 6 6 0
8 0
processpl 1000 7563 0 60 10 2 8 9 0
8 0
zombiepl 144 7503 0 0 1986 1986 0 1 0
8 0
ucredpl 96 4864 0 33 1 0 1 1 0
8 0
pgrppl 48 2202 0 26 1 0 1 1 0
8 0
sessionpl 144 1934 0 21 1 0 1 1 0
8 0
lockfspl 48 19 0 3 1 0 1 1 0
8 0
lockfpl 104 35 0 3 1 0 1 1 0
8 0
filepl 120 141717 0 159 24 19 5 7 0
8 0
fdescpl 432 7543 0 40 7 2 5 6 0
8 0
pipepl 304 5486 0 1 981 980 1 2 0
8 0
kqueuepl 184 2025 0 37 3 1 2 3 0
8 0
knotepl 120 2269229 0 122 6 0 6 6 0
8 0
futexpl 64 11264 0 1 3 2 1 1 0
8 0
sigapl 424 7563 0 60 12 5 7 8 0
8 0
plimitpl 152 118 0 21 1 0 1 1 0
8 0
scxspl 216 129318 0 0 5141 5140 1 2 0
8 1
acpiwqpl 32 20394 0 0 1 0 1 1 1
8 1
pfiaddrpl 120 8 0 4 2 1 1 1 0
8 0
xhcixfer 280 136083 0 2 1 0 1 1 0
8 0
percpumem 48 32 0 32 1 0 1 1 0
8 0
namei 1024 303750 0 0 6720 6719 1 1 0
8 1
vnodes 224 20958 0 20958 1233 0 1233 1233 0
8 0
uvmvnodes 80 20958 0 20958 428 0 428 428 0
8 0
nchpl 144 65490 0 4062 153 2 151 151 0
8 0
ffsino 240 20923 0 18761 1114 10 1104 1104 0
8 0
dino2pl 256 20923 0 18761 1184 11 1173 1173 0
8 0
dirhash 1024 195 0 155 20 0 20 20 0
8 0
art_node 16 2003 0 98 1 0 1 1 0
8 0
art_table 32 6534 0 120 925 924 1 2 0
8 0
art_heap4 256 6533 0 119 16 8 8 9 0
8 0
art_heap8 4096 1 0 1 1 0 1 1 0
8 0
pfrule 1360 26 0 5 3 2 1 2 0
8 0
pfstate 320 1257754 0 3978 1361 894 467 564 0
8 0
pfstkey 112 2491021 0 7943 725 380 345 385 0
8 0
pfstitem 24 2491021 0 7943 128 47 81 82 0
8 0
pfrktable 1344 16 0 5 2 1 1 1 0
8 0
pfrke_plain 168 34 0 17 2 1 1 1 0
8 0
pfosfpen 112 2142 0 714 42 21 21 21 0
8 0
pfosfp 40 2142 0 423 10 5 5 5 0
8 0
pffrent 40 374 0 0 70 70 0 1 0
8 0
pffrnode 88 187 0 0 70 70 0 1 0
8 0
pffrag 232 187 0 0 70 70 0 1 0
482 0
rttmr 72 1318 0 0 287 286 1 1 0
8 1
inpcb 312 22180 0 19 2 0 2 2 0
8 0
arp 88 897 0 61 2 0 2 2 0
8 0
tcpcb 736 2090 0 9 2 0 2 2 0
8 0
tcpqe 32 528 0 0 296 296 0 1 0
8 0
sackhl 24 55 0 0 48 48 0 1 0
8 0
syncache 296 3904 0 2 1425 1424 1 1 0
8 0
unpcb 136 7828 0 69 3 0 3 3 0
8 0
rtentry 112 2003 0 98 4 0 4 4 0
8 0
rtpcb 120 32 0 5 1 0 1 1 0
8 0
plcache 128 66 0 66 3 0 3 3 0
8 0
In use 37743K, total allocated 42768K; utilization 88.3%
(Under Failure Conditions)
# vmstat -m
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 1231 561 14450 1280 0
32 2399 673 179466 640 29
64 2932 76 32350 320 0
128 21770 22 700511 160 163
256 418 14 10974 80 0
512 115 5 5483 40 0
1024 101 111 24567 20 16658
2048 1597 13 5615 10 1751
4096 1065 2 13164 5 0
8192 59 2 473 5 0
16384 9 0 439 5 0
32768 8 0 47 5 0
65536 1 0 5661 5 0
131072 2 0 2 5 0
262144 1 0 1 5 0
524288 3 0 3 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, rtable, UFS mount, dirhash, ACPI, proc, exec, VM
swap,
UVM amap, UVM aobj, USB, USB device, temp
32 devbuf, pcb, rtable, ifaddr, sysctl, UFS mount, sem, dirhash,
ACPI,
in_multi, ether_multi, exec, UVM amap, USB, USB device, NDP, temp
64 devbuf, rtable, ifaddr, counters, vnodes, dirhash, ACPI, proc,
in_multi, VM swap, UVM amap, USB, USB device, temp
128 devbuf, rtable, ifaddr, sysctl, counters, vnodes, UFS mount, sem,
dirhash, ACPI, NFS srvsock, in_multi, UVM amap, USB, USB
device, temp
256 devbuf, rtable, ifaddr, counters, ioctlops, iov, vnodes, UFS
mount,
shm, VM map, dirhash, ACPI, exec, tdb, UVM amap, USB, USB device,
temp
512 devbuf, counters, ioctlops, UFS mount, dirhash, ACPI, file
desc, ttys,
newblk, USB HC, temp, kqueue
1024 devbuf, pcb, sysctl, counters, ioctlops, mount, shm, ACPI,
proc, ttys,
exec, UVM amap, USB device, crypto data, temp
2048 devbuf, ioctlops, iov, UFS mount, ACPI, VM swap, UVM aobj, USB,
USB device, temp, kqueue
4096 devbuf, ifaddr, counters, ioctlops, UFS mount, proc, USB, USB
device,
memdesc, temp
8192 devbuf, counters, proc, ttys, pagedep, UVM amap, temp, SYN cache
16384 devbuf, NFS daemon, MSDOSFS mount, ttys, temp
32768 devbuf, UFS quota, UFS mount, ISOFS mount, inodedep
65536 devbuf, temp
131072 devbuf
262144 devbuf
524288 VM swap, temp
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Size(s)
devbuf 6187 8602K 8603K 78644K 7995 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144
pcb 13 9K 9K 78644K 13 0 16,32,1024
rtable 221 9K 10K 78644K 4967 0 16,32,64,128,256
ifaddr 59 11K 12K 78644K 106 0
32,64,128,256,4096
sysctl 3 2K 2K 78644K 3 0 32,128,1024
counters 124 102K 102K 78644K 124 0
64,128,256,512,1024,4096,8192
ioctlops 0 0K 4K 78644K 2256 0
256,512,1024,2048,4096
iov 0 0K 2K 78644K 27 0 256,2048
mount 5 5K 5K 78644K 5 0 1024
vnodes 1249 79K 79K 78644K 1293 0 64,128,256
UFS quota 1 32K 32K 78644K 1 0 32768
UFS mount 21 59K 59K 78644K 21 0
16,32,128,256,512,2048,4096,32768
shm 2 2K 2K 78644K 2 0 256,1024
VM map 2 1K 1K 78644K 2 0 256
sem 2 1K 1K 78644K 2 0 32,128
dirhash 126 24K 24K 78644K 153 0
16,32,64,128,256,512
ACPI 22718 2526K 2553K 78644K 640288 0
16,32,64,128,256,512,1024,2048
file desc 1 1K 1K 78644K 1 0 512
proc 117 98K 114K 78644K 1591 0
16,64,1024,4096,8192
NFS srvsock 1 1K 1K 78644K 1 0 128
NFS daemon 1 16K 16K 78644K 1 0 16384
in_multi 13 1K 1K 78644K 13 0 32,64,128
ether_multi 3 1K 1K 78644K 3 0 32
ISOFS mount 1 32K 32K 78644K 1 0 32768
MSDOSFS mount 1 16K 16K 78644K 1 0 16384
ttys 31 149K 149K 78644K 31 0
512,1024,8192,16384
exec 0 0K 3K 78644K 9232 0 16,32,256,1024
tdb 3 1K 1K 78644K 3 0 256
pagedep 1 8K 8K 78644K 1 0 8192
inodedep 1 32K 32K 78644K 1 0 32768
newblk 1 1K 1K 78644K 1 0 512
VM swap 7 587K 587K 78644K 7 0 16,64,2048,524288
UVM amap 651 78K 78K 78644K 45648 0
16,32,64,128,256,1024,8192
UVM aobj 3 3K 3K 78644K 3 0 16,2048
USB 39 30K 30K 78644K 46 0
16,32,64,128,256,2048,4096
USB device 16 13K 13K 78644K 16 0
16,32,64,128,256,1024,2048,4096
USB HC 1 1K 1K 78644K 1 0 512
memdesc 1 4K 4K 78644K 1 0 4096
crypto data 1 1K 1K 78644K 1 0 1024
NDP 7 1K 1K 78644K 7 0 32
temp 30 4680K 4748K 78644K 276793 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,65536,524288
kqueue 46 77K 101K 78644K 2543 0 512,2048
SYN cache 2 16K 16K 78644K 2 0 8192
Memory Totals: In Use Free Requests
17296K 205K 993207
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg
Maxpg Idle
phpool 112 21179 0 5984 180 9 171 171 0
8 0
extentpl 40 113 0 47 1 0 1 1 0
8 0
pmappl 216 8403 0 37 3 0 3 3 0
8 0
pvpl 32 8371472 0 17686 1971 1815 156 1444 0
265 0
pdppl 4096 16815 0 83 7122 7039 83 101 0
8 0
rwobjpl 24 729123 0 23087 171 30 141 141 0
8 0
vmsppl 272 8403 0 37 4 1 3 4 0
8 0
vmmpepl 168 2028201 0 3916 1740 1563 177 364 0
357 0
vmmpekpl 168 1306042 0 30 2 0 2 2 0
8 0
uaddr 24 8404 0 38 1 0 1 1 0
8 0
uaddrbest 32 2 0 2 1 0 1 1 0
8 0
uaddrrnd 24 8404 0 38 1 0 1 1 0
8 0
aobjpl 72 2 0 2 1 0 1 1 0
8 0
dma16 16 29 0 0 1 1 0 1 0
8 0
dma32 32 9 0 0 1 1 0 1 0
8 0
dma64 64 10 0 0 1 1 0 1 0
8 0
dma128 128 1 0 0 1 1 0 1 0
8 0
dma256 256 10 0 0 1 1 0 1 0
8 0
dma512 512 7 0 1 1 0 1 1 0
8 0
dma4096 4096 1 0 0 1 1 0 1 0
8 0
amappl 88 34125 0 389 30 21 9 23 0
92 0
amappl1 80 765465 0 1274 120 91 29 42 0
8 0
amappl2 88 36222 0 241 50 42 8 30 0
8 0
amappl3 96 15214 0 81 9 6 3 9 0
8 0
amappl4 104 44739 0 63 29 26 3 24 0
8 0
amappl5 112 7379 0 33 16 14 2 3 0
8 0
amappl6 120 12570 0 50 5 2 3 4 0
8 0
amappl7 128 1000 0 13 2 1 1 2 0
8 0
amappl8 136 6266 0 20 13 12 1 13 0
8 0
amappl9 144 147 0 3 1 0 1 1 0
8 0
amappl10 152 43 0 2 1 0 1 1 0
8 0
amappl11 160 1006 0 44 9 7 2 3 0
8 0
amappl12 168 188 0 0 9 9 0 3 0
8 0
amappl13 176 215 0 6 13 12 1 1 0
8 0
amappl14 184 54 0 0 6 6 0 1 0
8 0
amappl15 192 5329 0 16 3 2 1 2 0
8 0
amappl16 200 1178 0 12 19 18 1 17 0
8 0
amapchunkpl 152 86015 0 985 202 163 39 114 0
158 0
anonpl 24 1680719 0 8977 384 327 57 268 0
1519 0
bufpl 288 58723 0 47219 3375 2 3373 3373 0
8 0
mbufpl 256 148399388 0 2750 679 501 178 178 0
8 0
mcl2k 2048 48291 0 22 22 12 10 15 0
8 0
mcl2k2 2112 144845009 0 2473 734 561 173 173 0
8 0
mcl4k 4096 3065 0 0 42 37 5 7 0
8 0
mcl8k 8192 719 0 0 5 0 5 5 0
8 0
mcl9k 9216 65 0 0 2 0 2 2 0
8 0
mcl12k 12288 41 0 0 2 0 2 2 0
8 0
mcl16k 16384 73 0 0 2 0 2 2 0
8 0
mcl64k 65536 8 0 0 1 0 1 1 0
8 0
sockpl 448 34420 0 84 35 24 11 13 0
8 0
procpl 672 8473 0 61 6 0 6 6 0
8 0
processpl 1000 8424 0 58 10 2 8 9 0
8 0
zombiepl 144 8366 0 0 2179 2179 0 1 0
8 0
ucredpl 96 6059 0 33 1 0 1 1 0
8 0
pgrppl 48 2537 0 24 1 0 1 1 0
8 0
sessionpl 144 2255 0 20 1 0 1 1 0
8 0
lockfspl 48 19 0 3 1 0 1 1 0
8 0
lockfpl 104 35 0 3 1 0 1 1 0
8 0
filepl 120 159683 0 148 25 20 5 7 0
8 0
fdescpl 432 8404 0 38 7 2 5 6 0
8 0
pipepl 304 6059 0 0 1087 1087 0 2 0
8 0
kqueuepl 184 2520 0 36 3 1 2 3 0
8 0
knotepl 120 2539217 0 109 6 0 6 6 0
8 0
futexpl 64 11825 0 1 3 2 1 1 0
8 0
sigapl 424 8424 0 58 12 5 7 8 0
8 0
plimitpl 152 126 0 21 1 0 1 1 0
8 0
scxspl 216 137838 0 0 5606 5605 1 2 0
8 1
acpiwqpl 32 22483 0 0 1 0 1 1 1
8 1
pfiaddrpl 120 8 0 4 2 1 1 1 0
8 0
xhcixfer 280 150007 0 2 1 0 1 1 0
8 0
percpumem 48 32 0 32 1 0 1 1 0
8 0
namei 1024 323569 0 0 7545 7544 1 1 0
8 1
vnodes 224 20958 0 20958 1233 0 1233 1233 0
8 0
uvmvnodes 80 20958 0 20958 428 0 428 428 0
8 0
nchpl 144 65496 0 4064 153 2 151 151 0
8 0
ffsino 240 20923 0 18761 1114 10 1104 1104 0
8 0
dino2pl 256 20923 0 18761 1184 11 1173 1173 0
8 0
dirhash 1024 195 0 155 20 0 20 20 0
8 0
art_node 16 2225 0 96 1 0 1 1 0
8 0
art_table 32 7341 0 120 994 992 2 2 0
8 0
art_heap4 256 7340 0 119 16 8 8 9 0
8 0
art_heap8 4096 1 0 1 1 0 1 1 0
8 0
pfrule 1360 26 0 5 3 2 1 2 0
8 0
pfstate 320 1547969 0 5661 1542 1026 516 564 0
8 1
pfstkey 112 3068375 0 11295 785 424 361 386 0
8 0
pfstitem 24 3068375 0 11295 132 53 79 82 0
8 0
pfrktable 1344 16 0 5 2 1 1 1 0
8 0
pfrke_plain 168 34 0 17 2 1 1 1 0
8 0
pfosfpen 112 2142 0 714 42 21 21 21 0
8 0
pfosfp 40 2142 0 423 10 5 5 5 0
8 0
pffrent 40 404 0 0 75 75 0 1 0
8 0
pffrnode 88 202 0 0 75 75 0 1 0
8 0
pffrag 232 202 0 0 75 75 0 1 0
482 0
rttmr 72 1490 0 0 350 350 0 1 0
8 0
inpcb 312 24892 0 12 3 1 2 3 0
8 0
arp 88 985 0 59 2 0 2 2 0
8 0
tcpcb 736 2429 0 6 3 1 2 3 0
8 0
tcpqe 32 762 0 0 328 328 0 1 0
8 0
sackhl 24 68 0 0 60 60 0 1 0
8 0
syncache 296 4540 0 0 1545 1545 0 1 0
8 0
unpcb 136 9496 0 67 3 0 3 3 0
8 0
rtentry 112 2225 0 96 4 0 4 4 0
8 0
rtpcb 120 32 0 5 1 0 1 1 0
8 0
plcache 128 66 0 66 3 0 3 3 0
8 0
In use 42133K, total allocated 46200K; utilization 91.2%
