On Wednesday, July 03, 2019, at 7:02 PM, Richard Elling wrote:
> This is strange. What does free(1) say about the distribution of memory?
Well, on FreeBSD there is no free (that sounds strange :). :(
I'm not sure the exact details on OS version 11.2 (which this machine runs,
I've already tried latest 12 with the same effects), but (at least part of) the
ZFS kernel memory is allocated with UMA:
vfs.zfs.zio.use_uma: 1
and vmstat -z reports UMA usage:
vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 384, 0, 249, 1, 249, 0, 0
UMA Zones: 3712, 0, 266, 0, 266, 0, 0
UMA Slabs: 80, 0, 1422803, 1054197, 7430054, 0, 0
UMA Hash: 256, 0, 53, 67, 110, 0, 0
4 Bucket: 32, 0, 5327, 6423, 1545566, 0, 0
6 Bucket: 48, 0, 1214, 10489, 828006, 0, 0
8 Bucket: 64, 0, 321, 8545, 365753, 11, 0
12 Bucket: 96, 0, 653, 8490, 419214, 0, 0
16 Bucket: 128, 0, 797, 9743, 483560, 0, 0
32 Bucket: 256, 0, 6702, 13398, 2050273, 133, 0
64 Bucket: 512, 0, 4150, 2178, 913650,12769, 0
128 Bucket: 1024, 0, 2071, 13389, 1113521,5780, 0
256 Bucket: 2048, 0, 128113, 286993,20651289, 374, 0
vmem btag: 56, 0, 350419, 5078, 795034,2505, 0
VM OBJECT: 240, 0, 402285, 7139, 4098869, 0, 0
RADIX NODE: 144, 0, 645045, 3927,22395703, 0, 0
MAP: 240, 0, 3, 61, 3, 0, 0
KMAP ENTRY: 128, 0, 10, 641, 10, 0, 0
MAP ENTRY: 128, 0, 28149, 3874, 9594907, 0, 0
VMSPACE: 2520, 0, 851, 100, 67669, 0, 0
fakepg: 104, 0, 15, 3519, 19558, 0, 0
mt_zone: 16400, 0, 400, 0, 400, 0, 0
16: 16, 0, 1350875, 6533,32479711, 0, 0
32: 32, 0, 240971, 11779,119606880, 0, 0
64: 64, 0, 1197721, 257543,195586627, 0, 0
128: 128, 0, 96231, 20019,88241754, 0, 0
256: 256, 0, 556369, 18011,125134147, 0, 0
512: 512, 0, 3669, 22891,72244916, 0, 0
1024: 1024, 0, 60070, 1554,48277366, 0, 0
2048: 2048, 0, 1171, 1009,68476018, 0, 0
4096: 4096, 0, 206254, 497, 656807, 0, 0
8192: 8192, 0, 454, 108, 1234689, 0, 0
16384: 16384, 0, 592, 98, 196964, 0, 0
32768: 32768, 0, 754, 48, 16782, 0, 0
65536: 65536, 0, 110, 44, 308413, 0, 0
64 pcpu: 8, 0, 3738, 2662, 3958, 0, 0
SLEEPQUEUE: 80, 0, 15787, 1604, 15787, 0, 0
Files: 80, 0, 4777, 5223,43348309, 0, 0
filedesc0: 1104, 0, 885, 276, 67757, 0, 0
rl_entry: 40, 0, 1337, 4763, 1337, 0, 0
TURNSTILE: 136, 0, 15787, 573, 15787, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
umtx_shm: 88, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1424, 0, 887, 257, 67756, 0, 0
THREAD: 1552, 0, 15706, 80, 31446, 0, 0
cpuset: 96, 0, 14320, 1957, 29973, 0, 0
audit_record: 1248, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 3355455, 49146, 50536,254888158, 0, 0
mbuf: 256, 3355455, 2, 15696,375783899, 0, 0
mbuf_cluster: 2048, 524288, 99682, 46, 99682, 0, 0
mbuf_jumbo_page: 4096, 262144, 0, 5359,34923585, 0, 0
mbuf_jumbo_9k: 9216, 77672, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 43690, 0, 0, 0, 0, 0
DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0
g_bio: 376, 0, 4, 2066,137080859, 0, 0
ttyinq: 160, 0, 300, 800, 795, 0, 0
ttyoutq: 256, 0, 157, 758, 415, 0, 0
FPU_save_area: 832, 0, 0, 0, 0, 0, 0
nvme_request: 128, 0, 0, 0, 0, 0, 0
cryptop: 88, 0, 0, 0, 0, 0, 0
cryptodesc: 72, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0
VNODE: 472, 0, 536859, 1789, 1753373, 0, 0
VNODEPOLL: 120, 0, 0, 0, 0, 0, 0
BUF TRIE: 144, 0, 925, 105023, 957833, 0, 0
NAMEI: 1024, 0, 0, 832,46981027, 0, 0
rentr: 24, 0, 0, 0, 0, 0, 0
S VFS Cache: 108, 0, 401236, 8789, 3361937, 0, 0
STS VFS Cache: 148, 0, 5006, 1910, 4574773, 0, 0
L VFS Cache: 328, 0, 12, 516, 27, 0, 0
LTS VFS Cache: 368, 0, 1, 29, 1, 0, 0
NCLNODE: 576, 0, 1793, 1147, 3365, 0, 0
DIRHASH: 1024, 0, 4, 24, 4, 0, 0
AIO: 208, 0, 374, 1108, 374, 0, 0
AIOP: 32, 0, 4, 3121, 46, 0, 0
AIOCB: 752, 0, 0, 325, 1246, 0, 0
AIOLIO: 280, 0, 0, 0, 0, 0, 0
pipe: 760, 0, 145, 390, 60770, 0, 0
procdesc: 136, 0, 0, 0, 0, 0, 0
Mountpoints: 816, 0, 57, 233, 58, 0, 0
ksiginfo: 112, 0, 1129, 2511, 6961, 0, 0
itimer: 352, 0, 0, 0, 0, 0, 0
KNOTE: 128, 0, 4538, 4979,154787502, 0, 0
socket: 864, 2092040, 3117, 3087, 3280916, 0, 0
unpcb: 240, 2092048, 1617, 703, 10846, 0, 0
ipq: 56, 16401, 0, 284, 1135, 0, 0
udp_inpcb: 464, 2092040, 9, 807, 1078463, 0, 0
udpcb: 32, 2092125, 9, 6116, 1078463, 0, 0
tcp_inpcb: 464, 2092040, 4136, 2160, 2191596, 0, 0
tcpcb: 1040, 2092041, 1490, 3037, 2191596, 0, 0
tcptw: 88, 27810, 2646, 2259, 534815, 0, 0
syncache: 168, 15364, 0, 1909, 1649126, 0, 0
hostcache: 128, 15376, 56, 1091, 59, 0, 0
sackhole: 32, 0, 0, 375, 1, 0, 0
tcpreass: 40, 32800, 0, 0, 0, 0, 0
udplite_inpcb: 464, 2092040, 0, 0, 0, 0, 0
ripcb: 464, 2092040, 0, 0, 0, 0, 0
IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0
rtentry: 208, 0, 25, 602, 85, 0, 0
selfd: 64, 0, 162, 6472, 7565181, 0, 0
swpctrie: 144, 8153163, 0, 0, 0, 0, 0
swblk: 136, 8153147, 0, 0, 0, 0, 0
md0: 512, 0, 4679, 113, 4679, 0, 0
FFS inode: 152, 0, 585, 1391, 14001, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 585, 1380, 14001, 0, 0
md1: 512, 0, 2936, 464, 3657, 0, 0
taskq_zone: 48, 0, 0, 8134,21789622, 0, 0
range_seg_cache: 64, 0,299873262,31775340,5522810684, 0, 0
metaslab_alloc_trace_cache: 64, 0, 21, 9651,53390222, 0, 0
zio_cache: 1040, 0, 1306, 38846,311152736, 0, 0
zio_link_cache: 48, 0, 16, 42314,303003224, 0, 0
zio_buf_512: 512, 0, 1019522, 1678, 7632038, 0, 0
zio_data_buf_512: 512, 0, 48, 856, 1188706, 0, 0
zio_buf_1024: 1024, 0, 99, 745, 1596915, 0, 0
zio_data_buf_1024: 1024, 0, 5, 479, 12047, 0, 0
zio_buf_1536: 1536, 0, 78, 346, 2441086, 0, 0
zio_data_buf_1536: 1536, 0, 0, 306, 14397, 0, 0
zio_buf_2048: 2048, 0, 94, 172, 1519330, 0, 0
zio_data_buf_2048: 2048, 0, 14, 232, 45897, 0, 0
zio_buf_2560: 2560, 0, 1, 204, 1548609, 0, 0
zio_data_buf_2560: 2560, 0, 35, 394, 169730, 0, 0
zio_buf_3072: 3072, 0, 0, 249,10786754, 0, 0
zio_data_buf_3072: 3072, 0, 46, 398, 190480, 0, 0
zio_buf_3584: 3584, 0, 3, 487,19050435, 0, 0
zio_data_buf_3584: 3584, 0, 22, 217, 58964, 0, 0
zio_buf_4096: 4096, 0, 35099, 11906,24904024, 0, 0
zio_data_buf_4096: 4096, 0, 1, 160, 10963, 0, 0
zio_buf_5120: 5120, 0, 13, 90, 424122, 0, 0
zio_data_buf_5120: 5120, 0, 5, 63, 11176, 0, 0
zio_buf_6144: 6144, 0, 18, 36, 678619, 0, 0
zio_data_buf_6144: 6144, 0, 1, 51, 4685, 0, 0
zio_buf_7168: 7168, 0, 6, 72, 1067404, 0, 0
zio_data_buf_7168: 7168, 0, 3, 57, 5698, 0, 0
zio_buf_8192: 8192, 0, 1, 118, 3570978, 0, 0
zio_data_buf_8192: 8192, 0, 2, 189, 13540, 0, 0
zio_buf_10240: 10240, 0, 0, 92, 850099, 0, 0
zio_data_buf_10240: 10240, 0, 4, 58, 9578, 0, 0
zio_buf_12288: 12288, 0, 0, 106, 874770, 0, 0
zio_data_buf_12288: 12288, 0, 3, 61, 15294, 0, 0
zio_buf_14336: 14336, 0, 0, 95, 378702, 0, 0
zio_data_buf_14336: 14336, 0, 13, 113, 49841, 0, 0
zio_buf_16384: 16384, 0, 282536, 14598,33665531, 0, 0
zio_data_buf_16384: 16384, 0, 2, 67, 42277, 0, 0
zio_buf_20480: 20480, 0, 0, 57, 483487, 0, 0
zio_data_buf_20480: 20480, 0, 30, 111, 71069, 0, 0
zio_buf_24576: 24576, 0, 0, 88, 495215, 0, 0
zio_data_buf_24576: 24576, 0, 26, 97, 44064, 0, 0
zio_buf_28672: 28672, 0, 0, 76, 251256, 0, 0
zio_data_buf_28672: 28672, 0, 9, 98, 24139, 0, 0
zio_buf_32768: 32768, 0, 0, 42, 535982, 0, 0
zio_data_buf_32768: 32768, 0, 7, 81, 20902, 0, 0
zio_buf_40960: 40960, 0, 1, 138, 3595448, 0, 0
zio_data_buf_40960: 40960, 0, 15, 68, 19566, 0, 0
zio_buf_49152: 49152, 0, 8, 39, 327378, 0, 0
zio_data_buf_49152: 49152, 0, 3, 77, 13534, 0, 0
zio_buf_57344: 57344, 0, 8, 41, 227512, 0, 0
zio_data_buf_57344: 57344, 0, 1, 82, 10946, 0, 0
zio_buf_65536: 65536, 0, 0, 40, 118811, 0, 0
zio_data_buf_65536: 65536, 0, 3, 43, 10900, 0, 0
zio_buf_81920: 81920, 0, 21, 32, 133108, 0, 0
zio_data_buf_81920: 81920, 0, 4, 133, 16542, 0, 0
zio_buf_98304: 98304, 0, 0, 36, 44456, 0, 0
zio_data_buf_98304: 98304, 0, 0, 61, 8186, 0, 0
zio_buf_114688: 114688, 0, 0, 38, 56197, 0, 0
zio_data_buf_114688: 114688, 0, 3, 135, 15126, 0, 0
zio_buf_131072: 131072, 0, 403, 7693, 7327234, 0, 0
zio_data_buf_131072: 131072, 0, 13, 1004, 209069, 0, 0
zio_buf_163840: 163840, 0, 0, 30, 1052, 0, 0
zio_data_buf_163840: 163840, 0, 0, 27, 296, 0, 0
zio_buf_196608: 196608, 0, 0, 33, 316, 0, 0
zio_data_buf_196608: 196608, 0, 0, 26, 218, 0, 0
zio_buf_229376: 229376, 0, 0, 30, 263, 0, 0
zio_data_buf_229376: 229376, 0, 0, 24, 202, 0, 0
zio_buf_262144: 262144, 0, 0, 28, 224, 0, 0
zio_data_buf_262144: 262144, 0, 0, 26, 129, 0, 0
zio_buf_327680: 327680, 0, 0, 29, 236, 0, 0
zio_data_buf_327680: 327680, 0, 0, 29, 212, 0, 0
zio_buf_393216: 393216, 0, 0, 27, 226, 0, 0
zio_data_buf_393216: 393216, 0, 0, 26, 140, 0, 0
zio_buf_458752: 458752, 0, 0, 33, 204, 0, 0
zio_data_buf_458752: 458752, 0, 0, 26, 80, 0, 0
zio_buf_524288: 524288, 0, 0, 26, 156, 0, 0
zio_data_buf_524288: 524288, 0, 0, 24, 77, 0, 0
zio_buf_655360: 655360, 0, 0, 27, 206, 0, 0
zio_data_buf_655360: 655360, 0, 0, 25, 116, 0, 0
zio_buf_786432: 786432, 0, 0, 31, 802, 0, 0
zio_data_buf_786432: 786432, 0, 0, 19, 70, 0, 0
zio_buf_917504: 917504, 0, 0, 32, 932, 0, 0
zio_data_buf_917504: 917504, 0, 0, 20, 46, 0, 0
zio_buf_1048576: 1048576, 0, 0, 34, 55615, 0, 0
zio_data_buf_1048576: 1048576, 0, 25, 27, 3488, 0, 0
zio_buf_1310720: 1310720, 0, 0, 0, 0, 0, 0
zio_data_buf_1310720: 1310720, 0, 0, 0, 0, 0, 0
zio_buf_1572864: 1572864, 0, 0, 0, 0, 0, 0
zio_data_buf_1572864: 1572864, 0, 0, 0, 0, 0, 0
zio_buf_1835008: 1835008, 0, 0, 0, 0, 0, 0
zio_data_buf_1835008: 1835008, 0, 0, 0, 0, 0, 0
zio_buf_2097152: 2097152, 0, 0, 0, 0, 0, 0
zio_data_buf_2097152: 2097152, 0, 0, 0, 0, 0, 0
zio_buf_2621440: 2621440, 0, 0, 0, 0, 0, 0
zio_data_buf_2621440: 2621440, 0, 0, 0, 0, 0, 0
zio_buf_3145728: 3145728, 0, 0, 0, 0, 0, 0
zio_data_buf_3145728: 3145728, 0, 0, 0, 0, 0, 0
zio_buf_3670016: 3670016, 0, 0, 0, 0, 0, 0
zio_data_buf_3670016: 3670016, 0, 0, 0, 0, 0, 0
zio_buf_4194304: 4194304, 0, 0, 0, 0, 0, 0
zio_data_buf_4194304: 4194304, 0, 0, 0, 0, 0, 0
zio_buf_5242880: 5242880, 0, 0, 0, 0, 0, 0
zio_data_buf_5242880: 5242880, 0, 0, 0, 0, 0, 0
zio_buf_6291456: 6291456, 0, 0, 0, 0, 0, 0
zio_data_buf_6291456: 6291456, 0, 0, 0, 0, 0, 0
zio_buf_7340032: 7340032, 0, 0, 0, 0, 0, 0
zio_data_buf_7340032: 7340032, 0, 0, 0, 0, 0, 0
zio_buf_8388608: 8388608, 0, 0, 0, 0, 0, 0
zio_data_buf_8388608: 8388608, 0, 0, 0, 0, 0, 0
zio_buf_10485760: 10485760, 0, 0, 0, 0, 0, 0
zio_data_buf_10485760: 10485760, 0, 0, 0, 0, 0, 0
zio_buf_12582912: 12582912, 0, 0, 0, 0, 0, 0
zio_data_buf_12582912: 12582912, 0, 0, 0, 0, 0, 0
zio_buf_14680064: 14680064, 0, 0, 0, 0, 0, 0
zio_data_buf_14680064: 14680064, 0, 0, 0, 0, 0, 0
zio_buf_16777216: 16777216, 0, 0, 0, 0, 0, 0
zio_data_buf_16777216: 16777216, 0, 0, 0, 0, 0, 0
lz4_ctx: 16384, 0, 0, 255,19326159, 0, 0
abd_chunk: 4096, 0, 420253, 19496,88679775, 0, 0
sa_cache: 144, 0, 534281, 3505, 1735475, 0, 0
dnode_t: 728, 0, 1414102, 763, 1739016, 0, 0
arc_buf_hdr_t_full: 224, 0, 1185506, 446732,589031125, 0, 0
arc_buf_hdr_t_l2only: 88, 0, 0, 0, 0, 0, 0
arc_buf_t: 64, 0, 318729, 14025,43073016, 0, 0
dmu_buf_impl_t: 232, 0, 1338113, 8236,24985458, 0, 0
zil_lwb_cache: 320, 0, 459, 11997, 5665648, 0, 0
zil_zcw_cache: 80, 0, 1, 3749, 6066228, 0, 0
zfs_znode_cache: 264, 0, 534281, 859, 1735475, 0, 0
TMPFS dirent: 64, 0, 101, 5975, 131, 0, 0
TMPFS node: 232, 0, 106, 1373, 132, 0, 0
TMPFS dirent: 64, 0, 66, 5266, 67, 0, 0
TMPFS node: 232, 0, 67, 1225, 68, 0, 0
TMPFS dirent: 64, 0, 1, 5579, 184, 0, 0
TMPFS node: 232, 0, 2, 1341, 185, 0, 0
pf mtags: 48, 0, 0, 830, 5, 0, 0
pf states: 296, 100009, 15826, 16518, 4222974, 0, 0
pf state keys: 88, 0, 15826, 18059, 4222974, 0, 0
pf source nodes: 136, 10005, 0, 0, 0, 0, 0
pf table entries: 160, 200000, 12, 63, 12, 0, 0
pf table counters: 64, 0, 0, 0, 0, 0, 0
pf frags: 112, 0, 0, 0, 0, 0, 0
pf frag entries: 40, 5000, 0, 0, 0, 0, 0
pf state scrubs: 40, 0, 0, 0, 0, 0, 0
top says on this machine:
Mem: 2371M Active, 21G Inact, 35G Wired, 52M Buf, 3923M Free
ARC: 7688M Total, 858M MFU, 4960M MRU, 9892K Anon, 273M Header, 1589M Other
1197M Compressed, 4630M Uncompressed, 3.87:1 Ratio
------------------------------------------
openzfs: openzfs-developer
Permalink:
https://openzfs.topicbox.com/groups/developer/T10533b84f9e1cfc5-Mcbe0070f704b59907a38125d
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
