Hi.
On 09.02.2012 02:29, Andriy Gapon wrote:
on 08/02/2012 12:31 Eugene M. Zheganin said the following:
Hi.
On 08.02.2012 02:17, Andriy Gapon wrote:
[output snipped]
Thank you. I don't see anything suspicious/unusual there.
Just case, do you have ZFS dedup enabled by a chance?
I think that examination of vmstat -m and vmstat -z outputs may provide some
clues as to what got all that memory wired.
Nope, I don't have deduplication feature enabled.
OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
I did. I didn't understand it, but kinda 'felt the atmosphere'. It was
pretty much similar to the output I supplied below. Most of the sizes
were used by 'solaris' and numerous 'zio' caches.
It could be very well possible that swap on zvol doesn't work well when the
kernel itself is starved on memory.
So I want to ask - how to report it and what should I include in such pr ?
I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be
ZFS-related. I suspect that you might be running into some kernel memory leak.
If you manage to reproduce the high wired value again, then vmstat -m and
vmstat -z may provide some useful information.
In this vein, do you use any out-of-tree kernel modules?
Also, can you try to monitor your system to see when wired count grows?
Nope, I don't have any 3rd party kernel modules.
Yes, I can monitor it, but I have no idea what should I exactly monitor.
This system is running squid with a dozens of authentication helpers,
freeradius + postgresql, sendmail and a perl squid log parser, which
uses postgresql too. net/isc-dhcp, quagga, net/mpd5, a bunch of sendmail
milters, net/samba35, bind. So it's some kind of a corporate production
zoo. As I write this letter, the wired amount of memory increases by 70
megs. Excuse me, 80 megs now.
The output I promised (if it's MORE acceptable in the form of a link to
a paste site, just say it):
[emz@taiga:etc/snmp]# vmstat -m
Type InUse MemUse HighUse Requests Size(s)
hhook 2 1K - 2 128
ithread 85 14K - 85 32,128,256
KTRACE 100 13K - 100 128
linker 280 226K - 384
16,32,64,128,256,512,1024,2048,4096
lockf 94 10K - 20264872 64,128
loginclass 3 1K - 367 64
pci_link 13 2K - 13 16,128
ip6ndp 55 5K - 78 64,128
ip6opt 23 6K - 142134 32,256
temp 146 20K - 114199
16,32,64,128,256,512,1024,2048,4096
devbuf 28285 56235K - 29225
16,32,64,128,256,512,1024,2048,4096
module 291 37K - 291 128
USBdev 39 10K - 39 64,128,512,1024
mtx_pool 2 16K - 2
USB 55 166K - 58 16,32,64,128,256,512,2048,4096
osd 22 1K - 10870 16,64
ddb_capture 1 48K - 1
subproc 831 1312K - 56233 512,4096
proc 2 16K - 2
session 66 9K - 16431 128
pgrp 73 10K - 16581 128
cred 650 102K - 818736 64,256
uidinfo 15 4K - 5420 128,2048
plimit 25 7K - 4948 256
kbdmux 8 18K - 8 16,512,1024,2048
sysctltmp 0 0K - 9741241 16,32,64,128,4096
sysctloid 4837 243K - 4950 16,32,64,128
sysctl 0 0K - 50230 16,32,64
tidhash 1 16K - 1
callout 3 1536K - 3
umtx 2712 339K - 2766 128
p1003.1b 1 1K - 1 16
SWAP 2 1097K - 2 64
bus-sc 84 686K - 2193
16,32,64,128,256,512,1024,2048,4096
bus 861 78K - 4641 16,32,64,128,256,512,1024
devstat 4 9K - 4 32,4096
eventhandler 83 7K - 83 64,128
kobj 194 776K - 231 4096
Per-cpu 1 1K - 1 32
aacbuf 241 72K - 273 64,128,512
rman 219 23K - 449 16,32,128
acpiintr 1 1K - 1 64
sbuf 1 1K - 967
16,32,64,128,256,512,1024,2048,4096
acpica 1641 174K - 50289
16,32,64,128,256,512,1024,2048,4096
DEVFS1 106 53K - 111 512
DEVFS3 261 66K - 269 256
stack 0 0K - 2 256
taskqueue 85 8K - 121 16,32,64,128,1024
Unitno 21 1K - 208557 32,64
DEVFS2 106 2K - 108 16
DEVFS_RULE 54 26K - 54 64,512
DEVFS 39 1K - 40 16,128
Witness 1 128K - 1
iov 0 0K - 12708587 16,32,64,128,256,512
select 1081 136K - 1108 128
ioctlops 0 0K - 5846280
16,32,64,128,256,512,1024,2048,4096
msg 4 30K - 4 2048,4096
sem 4 106K - 4 2048,4096
shm 11 40K - 13055 2048
tty 23 23K - 29 1024,2048
pts 5 2K - 9 256
mbuf_tag 133 13K - 425972406 32,64,128
shmfd 1 8K - 1
DEVFSP 8 1K - 8 64
pcb 618 176K - 1042950 16,32,64,128,1024,2048,4096
soname 221 27K - 70492663 16,32,128
acl 0 0K - 1575 4096
vfscache 1 2048K - 1
vfs_hash 1 1024K - 1
vnodes 6 1K - 17 64,256
vnodemarker 0 0K - 72943 512
mount 256 16K - 1498 16,32,64,128,256,512
BPF 65 74K - 66 128,512,4096
ether_multi 564 32K - 786 16,32,64
ifaddr 330 93K - 339 16,32,64,128,256,512,2048,4096
ifnet 34 67K - 37 128,256,512,2048
clone 8 32K - 8 4096
arpcom 23 1K - 23 16
gif 1 1K - 1 256
lltable 767 209K - 15188 256,512
vlan 103 8K - 465 64,128
routetbl 840 45K - 13455 32,64,128,256,512
igmp 33 9K - 34 256
CARP 32 13K - 32 64,256,1024
ipid 2 24K - 2
ip_moptions 2 1K - 2 64,256
in_multi 23 6K - 24 256
in_mfilter 1 1K - 1 1024
encap_export_host 1 1K - 1 1024
sctp_iter 0 0K - 29 256
sctp_ifn 16 2K - 16 128
sctp_ifa 44 6K - 44 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 29 16
hostcache 1 28K - 1
syncache 1 96K - 1
fragment 0 0K - 18 64,128
ip6_moptions 10 2K - 10 32,256
in6_multi 188 28K - 196 32,256
in6_mfilter 5 5K - 5 1024
mld 33 5K - 34 128
NLM 0 0K - 1 32
rpc 64 8K - 398 16,32,64,128,256,512,1024,2048
audit_evclass 179 6K - 218 32
freework 1 1K - 1 64
newblk 1 128K - 1
bmsafemap 1 8K - 1
inodedep 1 1024K - 1
pagedep 1 128K - 1
vm_pgdata 2 129K - 2 128
UMAHash 2 5K - 6 512,1024,2048,4096
NFSD srvcache 0 0K - 91 128
pfs_nodes 21 6K - 21 256
acpitask 1 2K - 1 2048
memdesc 1 4K - 1 4096
GEOM 61 12K - 297 16,32,64,128,256,512,1024,2048
atkbddev 2 1K - 2 64
ata_pci 1 1K - 1 64
CAM dev queue 3 1K - 3 128
CAM queue 11 1K - 56 16,32
acpisem 16 2K - 16 128
CAM SIM 3 1K - 3 256
scsi_cd 0 0K - 4 16
CAM periph 4 1K - 18 16,32,64,128,256
isadev 7 1K - 7 128
entropy 1024 64K - 1024 64
apmdev 1 1K - 1 128
CAM XPT 29 14K - 80 32,64,128,1024,2048
UART 3 2K - 3 16,512,1024
cdev 8 2K - 8 256
io_apic 1 2K - 1 2048
sigio 1 1K - 3 64
filedesc 544 566K - 60333
16,32,64,128,256,512,1024,2048,4096
msi 2 1K - 2 128
nexusdev 3 1K - 3 16
kenv 104 12K - 108 16,32,64,128
kqueue 24 39K - 405980 256,2048,4096
acpidev 27 2K - 27 64
proc-args 219 24K - 449753 16,32,64,128,256
solaris 427482 2357264K - 228638905
16,32,64,128,256,512,1024,2048,4096
kstat_data 4 1K - 4 64
netgraph_msg 0 0K - 17743 64,128,256,512,1024
netgraph_node 11 3K - 26 256
netgraph_hook 14 2K - 46 128
netgraph 3 1K - 9 64,256,512,1024
netgraph_sock 5 1K - 10 128
netgraph_path 0 0K - 8912 16,32
netgraph_mppc 0 0K - 3 1024
netgraph_l2tp 2 3K - 2 128,2048
netgraph_ksock 1 1K - 2 128
netgraph_iface 1 1K - 2 128
netgraph_ppp 1 12K - 2
[emz@taiga:etc/snmp]# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 199, 5, 199, 0, 0
UMA Zones: 896, 0, 199, 1, 199, 0, 0
UMA Slabs: 568, 0, 38730, 827, 2045963, 0, 0
UMA RCntSlabs: 568, 0, 2417, 152, 10161, 0, 0
UMA Hash: 256, 0, 79, 11, 81, 0, 0
16 Bucket: 152, 0, 18, 132, 156, 0, 0
32 Bucket: 280, 0, 35, 119, 299, 0, 0
64 Bucket: 536, 0, 54, 72, 644, 57, 0
128 Bucket: 1048, 0, 467, 262, 112841,14481, 0
VM OBJECT: 216, 0, 39763, 2267, 1794570, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 150412, 3666, 2131, 4810417, 0, 0
MAP ENTRY: 120, 0, 16746, 8457, 4147673, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 283, 77, 283, 0, 0
16: 16, 0, 65795, 565,103575464, 0, 0
32: 32, 0, 14388, 964,32512329, 0, 0
64: 64, 0, 164377, 14879,486891319, 0, 0
128: 128, 0, 17439, 2571,112770466, 0, 0
256: 256, 0, 12848, 15832,32654808, 0, 0
512: 512, 0, 149161, 4727, 6242680, 0, 0
1024: 1024, 0, 1970, 314, 151802, 0, 0
2048: 2048, 0, 1717, 349, 465184, 0, 0
4096: 4096, 0, 7621, 669, 375642, 0, 0
Files: 80, 0, 2210, 805,47573314, 0, 0
TURNSTILE: 136, 0, 1357, 83, 1384, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1160, 0, 259, 314, 55680, 0, 0
THREAD: 1112, 0, 864, 492, 53330, 0, 0
SLEEPQUEUE: 88, 0, 1357, 122, 1384, 0, 0
VMSPACE: 392, 0, 239, 681, 55662, 0, 0
cpuset: 72, 0, 2, 98, 2, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 2107, 709,137988001, 0, 0
mbuf: 256, 0, 58204, 1065,523117709, 0, 0
mbuf_cluster: 2048, 25600, 2816, 914, 7273137, 0, 0
mbuf_jumbo_page: 4096, 12800, 9, 543, 7683953, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 672, 29126, 0, 0
g_bio: 232, 0, 0, 752, 5733840, 0, 0
ttyinq: 160, 0, 240, 120, 555, 0, 0
ttyoutq: 256, 0, 127, 83, 293, 0, 0
ata_request: 328, 0, 0, 36, 14, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
taskq_zone: 48, 0, 0, 792, 69760, 0, 0
VNODE: 480, 0, 37315, 2269, 429291, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
S VFS Cache: 108, 0, 16761, 28350, 349225, 0, 0
L VFS Cache: 328, 0, 18, 1086, 13342, 0, 0
NAMEI: 1024, 0, 0, 384,29340521, 0, 0
NCLNODE: 560, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 0, 0, 0, 0, 0
zio_cache: 880, 0, 1, 1135,40366913, 0, 0
zio_link_cache: 48, 0, 0, 1440,11333105, 0, 0
zio_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0
zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0
sa_cache: 80, 0, 37248, 2262, 429178, 0, 0
dnode_t: 856, 0, 151801, 1919, 249658, 0, 0
dmu_buf_impl_t: 224, 0, 177337, 2404, 600337, 0, 0
arc_buf_hdr_t: 216, 0, 105487, 1253, 519289, 0, 0
arc_buf_t: 104, 0, 38391, 5205, 559830, 0, 0
zil_lwb_cache: 192, 0, 9, 611, 80899, 0, 0
zfs_znode_cache: 400, 0, 37248, 2325, 429178, 0, 0
pipe: 728, 0, 42, 328, 39330, 0, 0
Mountpoints: 768, 0, 20, 20, 20, 0, 0
ksiginfo: 112, 0, 564, 657, 29202, 0, 0
itimer: 344, 0, 1, 21, 1, 0, 0
KNOTE: 128, 0, 877, 776,90925709, 0, 0
pfsrctrpl: 152, 10000, 0, 0, 0, 0, 0
pfrulepl: 936, 0, 325, 11, 325, 0, 0
pfstatepl: 288, 10010, 3360, 2568, 1170450, 0, 0
pfstatekeypl: 288, 0, 3983, 2621, 1916481, 0, 0
pfstateitempl: 288, 0, 3983, 2439, 1439637, 0, 0
pfaltqpl: 240, 0, 0, 0, 0, 0, 0
pfpooladdrpl: 88, 0, 71, 55, 71, 0, 0
pfrktable: 1296, 1002, 6, 9, 11, 0, 0
pfrkentry: 160, 200016, 21, 27, 21, 0, 0
pffrent: 32, 5050, 0, 0, 0, 0, 0
pffrag: 80, 0, 0, 0, 0, 0, 0
pffrcache: 80, 10035, 0, 0, 0, 0, 0
pffrcent: 24, 50022, 0, 0, 0, 0, 0
pfstatescrub: 40, 0, 0, 0, 0, 0, 0
pfiaddrpl: 120, 0, 0, 0, 0, 0, 0
pfospfen: 112, 0, 700, 26, 700, 0, 0
pfosfp: 40, 0, 410, 94, 410, 0, 0
pfsync: 88, 0, 0, 0, 0, 0, 0
socket: 680, 25602, 1726, 596,25837811, 0, 0
ipq: 56, 819, 0, 126, 396, 0, 0
udp_inpcb: 392, 25600, 152, 728,20355207, 0, 0
udpcb: 16, 25704, 152, 856,20355207, 0, 0
tcp_inpcb: 392, 25600, 1782, 4968, 5284799, 0, 0
tcpcb: 976, 25600, 982, 762, 5284799, 0, 0
tcptw: 72, 5150, 800, 4350, 306578,97901, 0
syncache: 152, 15375, 0, 325, 5052872, 0, 0
hostcache: 136, 15372, 2053, 467, 7364, 0, 0
tcpreass: 40, 1680, 3, 585, 216370, 0, 0
sackhole: 32, 0, 0, 909, 55622, 0, 0
sctp_ep: 1368, 25600, 0, 0, 0, 0, 0
sctp_asoc: 2280, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 288, 43, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 25600, 6, 34, 8, 0, 0
unpcb: 240, 25600, 570, 550, 197771, 0, 0
rtentry: 200, 0, 756, 137, 1395, 0, 0
selfd: 56, 0, 1504, 701,705385407, 0, 0
SWAPMETA: 288, 116519, 4, 113, 866, 0, 0
NetGraph items: 72, 4118, 0, 812, 115049, 0, 0
NetGraph data items: 72, 522, 0, 493, 189271, 0, 0
P.S. after I pasted this, the wired memory gone down by 70 megs, the
final output (I guess enough is enough can be seen here:
http://dpaste.org/iA5mP/ ).
Thanks.
Eugene.
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[email protected]"
