Hi David,
Here are all the outputs you mentioned. Hopefully something will improve
our understanding of this story.
David Gwynne <da...@gwynne.id.au> writes:
> Ofails are the sum of output errors and queue drops. Can you figure
> out which one it is with netstat -I igc0 -e and netstat -I igc0 -d?
% netstat -I igc2 -d
Name Mtu Network Address Ipkts Idrop Opkts Odrop Colls
igc2 1500 <Link> 00:e2:69:53:c0:0b 18487201 0 53088501 172714 0
igc2 1500 192.168.172 192.168.172.1 18487201 0 53088501 172714 0
igc2 1500 192.168.172 192.168.172.53 18487201 0 53088501 172714 0
% netstat -I igc2 -e
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Colls
igc2 1500 <Link> 00:e2:69:53:c0:0b 18487201 0 53088501 0 0
igc2 1500 192.168.172 192.168.172.1 18487201 0 53088501 0 0
igc2 1500 192.168.172 192.168.172.53 18487201 0 53088501 0 0
> The state of the rxring accounting according to "systat mb" output
> would be interesting too.
IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
System mbufs 0 256 5561 479
mcl2k 2048 40 14
mcl2k2 2112 4534 442
mcl4k 4096 0 8
mcl8k 8192 0 6
mcl16k 16384 0 1
lo0
igc0 0 2048 24 10 1023 24
1 2048 14 10 1023 14
2 2048 15 10 1023 15
3 2048 12 10 1023 12
igc1 0 2048 27 10 1023 27
1 2048 27 10 1023 27
2 2048 24 10 1023 24
3 2048 25 10 1023 25
igc2 0 2048 18 10 1023 18
1 2048 14 10 1023 14
2 2048 17 10 1023 17
3 2048 17 10 1023 17
igc3 0 2048 30 10 1023 30
1 2048 31 10 1023 31
2 2048 33 10 1023 33
3 2048 30 10 1023 30
> kstat output is easy to get too, though I'm not sure it will be useful
> in this situation.
igc2:0:rxq:0
packets: 4544785 packets
bytes: 2087452079 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
igc2:0:rxq:1
packets: 5722952 packets
bytes: 3638339639 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
igc2:0:rxq:2
packets: 5479968 packets
bytes: 2818395627 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
igc2:0:rxq:3
packets: 2739496 packets
bytes: 1411808602 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
igc2:0:txq:0
packets: 19740629 packets
bytes: 24868676639 bytes
qdrops: 5 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 1023 packets
oactive: false
igc2:0:txq:1
packets: 11828063 packets
bytes: 14495415780 bytes
qdrops: 42113 packets
errors: 0 packets
qlen: 1023 packets
maxqlen: 1023 packets
oactive: false
igc2:0:txq:2
packets: 7975725 packets
bytes: 9745852229 bytes
qdrops: 95687 packets
errors: 0 packets
qlen: 1023 packets
maxqlen: 1023 packets
oactive: false
igc2:0:txq:3
packets: 13544084 packets
bytes: 16273238465 bytes
qdrops: 34909 packets
errors: 0 packets
qlen: 1023 packets
maxqlen: 1023 packets
oactive: false
> The mbuf (and all other) pool counters from vmstat -m are easy to get too.
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 1060 732 117924 1280 0
32 980 812 162242 640 4
64 2117 59 758707 320 4131
128 510540 52 25550193 160 637
256 225 287 63967 80 10123
512 395 29 39457 40 0
1024 125 7 163014 20 4
2048 16443 71 46030 10 21091
4096 58 10 83403 5 6680
8192 58 3 5746 5 0
16384 6 0 4153 5 0
32768 45 0 94 5 0
65536 2 0 136029 5 0
131072 8 0 8 5 0
262144 2 0 2 5 0
524288 2 0 2 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, rtable, ifaddr, UFS mount, dirhash, ACPI, proc, exec,
VM swap, UVM amap, UVM aobj, USB, USB device, temp, DRM
32 devbuf, pcb, rtable, ifaddr, sysctl, UFS mount, sem, dirhash, ACPI,
in_multi, ether_multi, exec, UVM amap, USB, USB device, NDP, temp,
AGP Memory, DRM
64 devbuf, rtable, ifaddr, counters, vnodes, UFS mount, dirhash, ACPI,
proc, in_multi, VM swap, UVM amap, USB, USB device, temp, DRM
128 devbuf, rtable, ifaddr, sysctl, counters, log, vnodes, UFS mount, sem,
dirhash, ACPI, NFS srvsock, in_multi, UVM amap, UVM aobj, USB,
USB device, ip6_options, temp, DRM
256 devbuf, rtable, ifaddr, counters, ioctlops, vnodes, shm, VM map,
dirhash, ACPI, exec, tdb, UVM amap, USB, USB device, temp, DRM
512 devbuf, counters, ioctlops, iov, UFS mount, dirhash, ACPI, file desc,
ttys, newblk, USB HC, temp, kqueue, DRM
1024 devbuf, pcb, sysctl, counters, ioctlops, mount, shm, dirhash, ACPI,
proc, ttys, exec, USB device, crypto data, temp, DRM
2048 devbuf, ioctlops, iov, UFS mount, ACPI, VM swap, UVM aobj, USB, temp,
kqueue, DRM
4096 devbuf, ifaddr, counters, ioctlops, iov, proc, ttys, USB, memdesc,
temp, DRM
8192 devbuf, counters, proc, ttys, pagedep, UVM amap, temp, SYN cache, DRM
16384 devbuf, NFS daemon, MSDOSFS mount, ttys, temp, DRM
32768 devbuf, UFS quota, UFS mount, ISOFS mount, inodedep
65536 devbuf, temp
131072 devbuf
262144 VM swap
524288 VM swap, temp
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Size(s)
devbuf 34299 37248K 37326K 78644K 60162 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072
pcb 13 9K 9K 78644K 15 0 16,32,1024
rtable 90 3K 3K 78644K 3206 0 16,32,64,128,256
ifaddr 79 13K 14K 78644K 101 0 16,32,64,128,256,4096
sysctl 3 2K 2K 78644K 7 0 32,128,1024
counters 82 68K 68K 78644K 82 0
64,128,256,512,1024,4096,8192
ioctlops 0 0K 4K 78644K 2570 0 256,512,1024,2048,4096
iov 0 0K 4K 78644K 24 0 512,2048,4096
mount 9 9K 9K 78644K 9 0 1024
log 0 0K 1K 78644K 2 0 128
vnodes 1253 80K 80K 78644K 3686 0 64,128,256
UFS quota 1 32K 32K 78644K 1 0 32768
UFS mount 37 73K 73K 78644K 37 0
16,32,64,128,512,2048,32768
shm 2 2K 2K 78644K 2 0 256,1024
VM map 2 1K 1K 78644K 2 0 256
sem 2 1K 1K 78644K 2 0 32,128
dirhash 651 124K 124K 78644K 651 0
16,32,64,128,256,512,1024
ACPI493824 61687K 61693K 78644K 25998426 0
16,32,64,128,256,512,1024,2048
file desc 2 1K 13K 78644K 1352 0 512
proc 101 117K 126K 78644K 3819 0 16,64,1024,4096,8192
NFS srvsock 1 1K 1K 78644K 1 0 128
NFS daemon 1 16K 16K 78644K 1 0 16384
in_multi 14 1K 1K 78644K 16 0 32,64,128
ether_multi 4 1K 1K 78644K 6 0 32
ISOFS mount 1 32K 32K 78644K 1 0 32768
MSDOSFS mount 1 16K 16K 78644K 1 0 16384
ttys 91 336K 336K 78644K 91 0
512,1024,4096,8192,16384
exec 0 0K 3K 78644K 66192 0 16,32,256,1024
tdb 3 1K 1K 78644K 3 0 256
pagedep 1 8K 8K 78644K 1 0 8192
inodedep 1 32K 32K 78644K 1 0 32768
newblk 1 1K 1K 78644K 1 0 512
VM swap 8 583K 585K 78644K 10 0
16,64,2048,262144,524288
UVM amap 637 40K 108K 78644K 296387 0 16,32,64,128,256,8192
UVM aobj 13 3K 3K 78644K 13 0 16,128,2048
USB 12 11K 42K 78644K 58 0
16,32,64,128,256,2048,4096
USB device 4 2K 3K 78644K 20 0 16,32,64,128,256,1024
USB HC 1 1K 1K 78644K 1 0 512
memdesc 1 4K 4K 78644K 1 0 4096
crypto data 1 1K 1K 78644K 1 0 1024
ip6_options 0 0K 1K 78644K 12 0 128
NDP 7 1K 1K 78644K 7 0 32
temp 30 4698K 4766K 78644K 675689 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,65536,524288
kqueue 31 47K 187K 78644K 17135 0 512,2048
SYN cache 2 16K 16K 78644K 2 0 8192
AGP Memory 1 1K 1K 78644K 1 0 32
DRM 750 208K 273K 78644K 1164 0
16,32,64,128,256,512,1024,2048,4096,8192,16384
Memory Totals: In Use Free Requests
105514K 347K 27130972
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
phpool 112 59934 0 14374 444 29 415 417 0 8 0
extentpl 40 84 0 39 1 0 1 1 0 8 0
pmappl 216 53111 0 38 191 188 3 7 0 8 0
pvpl 32 48485509 0 35427 25401 25085 316 1656 0 265 0
pdppl 4096 53111 0 38 21511 21473 38 109 0 8 0
rwobjpl 24 4086925 0 75428 1789 1332 457 473 0 8 0
vmsppl 272 53111 0 38 292 288 4 8 0 8 0
vmmpepl 168 13387471 0 5230 39937 39703 234 609 0 357 0
vmmpekpl 168 5598696 0 41 18 14 4 5 0 8 0
uaddr 24 53112 0 39 1 0 1 1 0 8 0
uaddrbest 32 2 0 2 1 0 1 1 0 8 0
uaddrrnd 24 53112 0 39 1 0 1 1 0 8 0
aobjpl 72 12 0 12 1 0 1 1 0 8 0
dma16 16 28 0 0 1 1 0 1 0 8 0
dma32 32 9 0 0 1 1 0 1 0 8 0
dma64 64 10 0 0 1 1 0 1 0 8 0
dma128 128 2 0 0 1 1 0 1 0 8 0
dma256 256 10 0 0 1 1 0 1 0 8 0
dma512 512 3 0 1 1 0 1 1 0 8 0
dma4096 4096 1 0 0 1 1 0 1 0 8 0
amappl 88 220963 0 384 62 51 11 28 0 92 0
amappl1 80 4436064 0 2231 4840 4788 52 138 0 8 0
amappl2 88 197059 0 406 307 295 12 35 0 8 0
amappl3 96 168559 0 158 152 147 5 10 0 8 0
amappl4 104 240036 0 255 70 62 8 29 0 8 0
amappl5 112 51608 0 73 122 119 3 4 0 8 0
amappl6 120 98261 0 84 47 43 4 5 0 8 0
amappl7 128 21325 0 46 90 88 2 4 0 8 0
amappl8 136 42799 0 39 24 21 3 13 0 8 0
amappl9 144 2481 0 28 4 2 2 2 0 8 0
amappl10 152 533 0 1 8 7 1 1 0 8 0
amappl11 160 7255 0 56 540 537 3 3 0 8 0
amappl12 168 544 0 4 15 14 1 3 0 8 0
amappl13 176 1316 0 10 1 0 1 1 0 8 0
amappl14 184 1338 0 0 37 37 0 11 0 8 0
amappl15 192 34615 0 30 92 90 2 4 0 8 0
amappl16 200 8892 0 12 81 79 2 23 0 8 0
amapchunkpl 152 652802 0 1205 547 495 52 150 0 158 0
anonpl 24 9333927 0 22094 2231 2083 148 371 0 364 0
bufpl 288 99448 0 65083 5349 345 5004 5294 0 8 0
mbufpl 256 87091916 0 5563 904 470 434 479 0 8 0
mtagpl 96 1056 0 0 1 0 1 1 0 8 0
mcl2k 2048 22094 0 40 28 14 14 14 0 8 0
mcl2k2 2112 81571681 0 4535 944 551 393 442 0 8 0
mcl4k 4096 9154 0 0 208 205 3 8 0 8 0
mcl8k 8192 6660 0 0 6 1 5 6 0 8 0
mcl16k 16384 1 0 0 1 0 1 1 0 8 0
sockpl 456 103654 0 67 801 791 10 23 0 8 0
srpgc 96 38 0 0 27 27 0 1 0 8 0
procpl 672 53538 0 86 331 323 8 14 0 8 0
processpl 1000 53158 0 85 501 489 12 20 0 8 0
zombiepl 144 53073 0 0 8782 8781 1 2 0 8 1
ucredpl 104 36478 0 24 74 73 1 2 0 8 0
pgrppl 48 10754 0 29 1 0 1 1 0 8 0
sessionpl 144 9266 0 24 129 127 2 3 0 8 0
lockfspl 48 1273 0 2 1 0 1 1 0 8 0
lockfpl 104 2544 0 2 1 0 1 1 0 8 0
filepl 120 715537 0 142 533 528 5 14 0 8 0
fdescpl 432 53112 0 39 465 459 6 13 0 8 0
pipepl 288 47280 0 3 290 289 1 6 0 8 0
kqueuepl 184 17113 0 21 148 146 2 5 0 8 0
knotepl 120 8674320 0 104 151 143 8 14 0 8 0
futexpl 64 8172 0 0 5 5 0 1 0 8 0
sigapl 424 53158 0 85 519 508 11 18 0 8 0
plimitpl 152 303 0 17 1 0 1 1 0 8 0
scxspl 216 467189 0 0 13731 13730 1 2 0 8 1
acpiwqpl 32 69450 0 0 1 0 1 1 1 8 1
idrpl 32 185 0 164 2 0 2 2 0 8 0
kstatmem 264 38 0 38 3 0 3 3 0 8 0
xhcixfer 280 134 0 1 1 0 1 1 0 8 0
percpumem 32 32 0 32 1 0 1 1 0 8 0
namei 1024 3112009 0 0 18008 18007 1 3 0 8 1
vnodes 216 72070 0 72070 4004 0 4004 4004 0 8 0
uvmvnodes 80 72070 0 72070 1471 0 1471 1471 0 8 0
nchpl 144 2048179 0 5925 220 0 220 220 0 8 0
ffsino 240 1577588 0 72051 4239 0 4239 4239 0 8 0
dino2pl 256 1577588 0 72051 4505 1 4504 4504 0 8 0
dirhash 1024 841 0 841 106 0 106 106 0 8 0
art_node 16 272 0 34 1 0 1 1 0 8 0
art_table 32 1245 0 111 15 13 2 2 0 8 0
art_heap4 256 1244 0 110 11 4 7 9 0 8 0
art_heap8 4096 1 0 1 1 0 1 1 0 8 0
rttmr 136 239 0 0 127 127 0 1 0 8 0
pfrule 1360 36 0 10 3 1 2 3 0 8 0
pfstate 336 883197 0 0 4137 4137 0 229 0 8 0
pfstkey 120 1155453 0 12 637 631 6 90 0 8 0
pfstitem 24 1155407 0 0 36 36 0 18 0 8 0
pfosfpen 112 2142 0 714 21 0 21 21 0 8 0
pfosfp 40 2142 0 423 5 0 5 5 0 8 0
inpcb 336 51285 0 27 325 322 3 7 0 8 0
arp 88 56 0 8 1 0 1 1 0 8 0
tcpcb 768 50746 0 13 633 631 2 7 0 8 0
tcpqe 32 3482 0 0 1433 1433 0 1 0 8 0
sackhl 24 136 0 0 93 93 0 1 0 8 0
syncache 296 11389 0 0 5744 5744 0 4 0 8 0
unpcb 144 52339 0 36 143 141 2 5 0 8 0
rtentry 112 273 0 38 2 0 2 2 0 8 0
rtpcb 120 30 0 4 1 0 1 1 0 8 0
plcache 128 44 0 44 2 0 2 2 0 8 0
drmsc 128 5 0 0 1 1 0 1 0 8 0
ictx 640 9 0 5 1 0 1 1 0 8 0
drmobj 1024 53 0 36 7 1 6 7 0 8 0
i915_request 640 12 0 4 1 0 1 1 0 8 0
gspri 64 8 0 0 1 1 0 1 0 8 0
drmvma 704 30 0 17 3 0 3 3 0 8 0
xapl 32 1 0 1 1 0 1 1 0 8 0
In use 93566K, total allocated 101628K; utilization 92.1%
