Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
On 20.11.2018 23:45, Mark Johnston wrote:
>> Where is other 11G of memory?!
>>
>> I've checked USED and FREE in "vmstat -z" output and got this:
>>
>> $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n |
>> tail -20
>> 23001088 9171456 MAP ENTRY
>> 29680800 8404320 VM OBJECT
>> 34417408 10813952 256
>> 36377964 2665656 S VFS Cache
>> 50377392 53856 sa_cache
>> 50593792 622985216 zio_buf_131072
>> 68913152 976896 mbuf_cluster
>> 73543680 7225344 mbuf_jumbo_page
>> 92358552 67848 zfs_znode_cache
>> 95731712 51761152 4096
>> 126962880 159581760 dmu_buf_impl_t
>> 150958080 233920512 mbuf_jumbo_9k
>> 165164600 92040 VNODE
>> 192701120 30350880 UMA Slabs
>> 205520896 291504128 zio_data_buf_1048576
>> 222822400 529530880 zio_data_buf_524288
>> 259143168 293476864 zio_buf_512
>> 352485376 377061376 zio_buf_16384
>> 376109552 346474128 dnode_t
>> 2943016960 5761941504 abd_chunk
>> $
>>
>> And total USED/FREE numbers is very strange for me:
>>
>> $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}'
>> 5717965420 9328951088
>> $ One more datapoint: I needed to reboot system and right after reboot
ARC becomes 14GiB (what was expected) and now it slows defalate, though
I didn't get more "Free" memory. So, ARC becomes smaller and smaller
after first fill-up, but all memory is "Wired".
--
// Lev Serebryakov
signature.asc
Description: OpenPGP digital signature
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
Hello Eugene, Wednesday, November 21, 2018, 5:19:32 PM, you wrote: >> It is more likely that there is a bug in "priority" for memory >> distribution: abd FREE memory has higher priority than ARC somehow. > OTOH, if "abd_chunks" part of ZFS ARC is not released to the UMA > from which it was allocated (to be reused by other parts of the kernel) > but is reused by ZFS itself, then top(1) should report it as part of ZFS ARC > and not as other "Wired" memory. It will be better to system use this memory for ARC and not as "Free" chunks for future usage :-) -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
21.11.2018 21:03, Lev Serebryakov wrote: >>> And here top consumers. I don't like that most of te memory is in FREE >>> abd_chunks - 8GB! >>> >>> USED (b) FREE (b) NAME >>> 531902464 8392536064 abd_chunk >> This may be a leak. Can you monitor this value for longer run? > No problem. > > But I don't think it is "true" leak, when memory is NEVER > freed, as uptime of this box is 11 days already, and sometimes ARC is much > larger than 2GiB. For example, now it is inflated to 8GiB. > > Ok, and now it is: > > USED (b) FREE (b) NAME > 7307493376 2383060992 abd_chunk > > It is more likely that there is a bug in "priority" for memory > distribution: abd FREE memory has higher priority than ARC somehow. OTOH, if "abd_chunks" part of ZFS ARC is not released to the UMA from which it was allocated (to be reused by other parts of the kernel) but is reused by ZFS itself, then top(1) should report it as part of ZFS ARC and not as other "Wired" memory. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
Hello Eugene, Wednesday, November 21, 2018, 4:23:29 PM, you wrote: >> And here top consumers. I don't like that most of te memory is in FREE >> abd_chunks - 8GB! >> >> USED (b) FREE (b) NAME >> 531902464 8392536064 abd_chunk > This may be a leak. Can you monitor this value for longer run? No problem. But I don't think it is "true" leak, when memory is NEVER freed, as uptime of this box is 11 days already, and sometimes ARC is much larger than 2GiB. For example, now it is inflated to 8GiB. Ok, and now it is: USED (b) FREE (b) NAME 35445924 3639276 S VFS Cache 37336376 1067704 arc_buf_hdr_t_l2only 47760912 2670336 sa_cache 51531120 55440 VM OBJECT 55574528 30408704 zio_data_buf_1048576 56183040 21365760 256 61734912 50331648 zio_buf_131072 70467584 2375680 mbuf_cluster 72028160 3895296 mbuf_jumbo_page 87561672 4856808 zfs_znode_cache 110448640 32768 4096 124137120 158133600 dmu_buf_impl_t 150976512 7188480 mbuf_jumbo_9k 156588360 8539896 VNODE 208290960 1040 UMA Slabs 247484416 280662016 zio_buf_512 275775488 96993280 zio_data_buf_524288 361406864 314169856 dnode_t 417366016 1015808 zio_buf_16384 7307493376 2383060992 abd_chunk It is more likely that there is a bug in "priority" for memory distribution: abd FREE memory has higher priority than ARC somehow. -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
21.11.2018 18:34, Lev Serebryakov wrote: > And here top consumers. I don't like that most of te memory is in FREE > abd_chunks - 8GB! > > USED (b) FREE (b) NAME > 531902464 8392536064 abd_chunk This may be a leak. Can you monitor this value for longer run? ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
Hello Lev, Wednesday, November 21, 2018, 1:56:18 PM, you wrote: >> Could you show the output of "vmstat -s" when in this state? > Now ARC is 1.9GB (!!!) and 15G is still Wired: > % vmstat -s > 1311134466 cpu context switches > 2126950424 device interrupts > 45414140 software interrupts > 103215017 traps > 2910067754 system calls >25 kernel threads created >108756 fork() calls > 50914 vfork() calls > 111 rfork() calls > 0 swap pager pageins > 0 swap pager pages paged in > 0 swap pager pageouts > 0 swap pager pages paged out > 4201 vnode pager pageins > 51719 vnode pager pages paged in > 31597 vnode pager pageouts > 63028 vnode pager pages paged out > 7 page daemon wakeups > 1212387275 pages examined by the page daemon > 0 clean page reclamation shortfalls > 0 pages reactivated by the page daemon > 8281789 copy-on-write faults > 7909 copy-on-write optimized faults > 0180 zero fill pages zeroed > 24762 zero fill pages prezeroed > 2 intransit blocking page faults > 63838217 total VM faults taken > 3816 page faults requiring I/O > 0 pages affected by kernel thread creation > 3996512 pages affected by fork() > 1842784 pages affected by vfork() > 5499 pages affected by rfork() > 88771375 pages freed > 0 pages freed by daemon > 21591789 pages freed by exiting processes > 4936 pages active >161234 pages inactive > 16674 pages in the laundry queue > 3848989 pages wired down > 39951 pages free > 4096 bytes per page > 612681203 total name lookups > cache hits (95% pos + 0% neg) system 0% per-directory > deletions 0%, falsehits 0%, toolong 0% > % And here top consumers. I don't like that most of te memory is in FREE abd_chunks - 8GB! USED (b) FREE (b) NAME 28785840 9299280 VM OBJECT 33817600 11413760 256 36598392 2486808 S VFS Cache 38157856 246224 arc_buf_hdr_t_l2only 44302336 629276672 zio_buf_131072 50377824 53424 sa_cache 72323072 85626880 mbuf_jumbo_page 74094592 55296 mbuf_cluster 90320896 63102976 4096 92359344 67056 zfs_znode_cache 126912480 159632160 dmu_buf_impl_t 150967296 233911296 mbuf_jumbo_9k 165164600 92040 VNODE 199124480 23927520 UMA Slabs 209715200 8388608 zio_data_buf_1048576 218103808 48758784 zio_data_buf_524288 259242496 293377536 zio_buf_512 346030080 383516672 zio_buf_16384 375592672 346991008 dnode_t 531902464 8392536064 abd_chunk -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
Hello Mark, Tuesday, November 20, 2018, 11:45:37 PM, you wrote: > Could you show the output of "vmstat -s" when in this state? Now ARC is 1.9GB (!!!) and 15G is still Wired: % vmstat -s 1311134466 cpu context switches 2126950424 device interrupts 45414140 software interrupts 103215017 traps 2910067754 system calls 25 kernel threads created 108756 fork() calls 50914 vfork() calls 111 rfork() calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 4201 vnode pager pageins 51719 vnode pager pages paged in 31597 vnode pager pageouts 63028 vnode pager pages paged out 7 page daemon wakeups 1212387275 pages examined by the page daemon 0 clean page reclamation shortfalls 0 pages reactivated by the page daemon 8281789 copy-on-write faults 7909 copy-on-write optimized faults 0180 zero fill pages zeroed 24762 zero fill pages prezeroed 2 intransit blocking page faults 63838217 total VM faults taken 3816 page faults requiring I/O 0 pages affected by kernel thread creation 3996512 pages affected by fork() 1842784 pages affected by vfork() 5499 pages affected by rfork() 88771375 pages freed 0 pages freed by daemon 21591789 pages freed by exiting processes 4936 pages active 161234 pages inactive 16674 pages in the laundry queue 3848989 pages wired down 39951 pages free 4096 bytes per page 612681203 total name lookups cache hits (95% pos + 0% neg) system 0% per-directory deletions 0%, falsehits 0%, toolong 0% % -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
On Tue, Nov 20, 2018 at 03:42:24PM +0300, Lev Serebryakov wrote:
>
> I have server which is mostly torrent box. It uses ZFS and equipped
> with 16GiB of physical memory. It is running 11-STABLE (r339914 now).
>
> I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago.
>
> I was used to see 13-14GiB of memory in ZFS ARC and it was Ok.
> Sometimes it "locks" under heavy disk load due to ARC memory pressure,
> but it was bearable, and as ZFS is main reason this server exists, I
> didn't limit ARC.
>
> But new revision (r339914) shows very strange behaivor: ARC is no more
> than 4GiB, but kernel has 15GiB wired:
>
> Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free
> ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other
> 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio
>
> It is typical numbers for last week: 15G wired, 237M Free, but only
> 4252M ARC!
>
> Where is other 11G of memory?!
>
> I've checked USED and FREE in "vmstat -z" output and got this:
>
> $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n |
> tail -20
> 23001088 9171456 MAP ENTRY
> 29680800 8404320 VM OBJECT
> 34417408 10813952 256
> 36377964 2665656 S VFS Cache
> 50377392 53856 sa_cache
> 50593792 622985216 zio_buf_131072
> 68913152 976896 mbuf_cluster
> 73543680 7225344 mbuf_jumbo_page
> 92358552 67848 zfs_znode_cache
> 95731712 51761152 4096
> 126962880 159581760 dmu_buf_impl_t
> 150958080 233920512 mbuf_jumbo_9k
> 165164600 92040 VNODE
> 192701120 30350880 UMA Slabs
> 205520896 291504128 zio_data_buf_1048576
> 222822400 529530880 zio_data_buf_524288
> 259143168 293476864 zio_buf_512
> 352485376 377061376 zio_buf_16384
> 376109552 346474128 dnode_t
> 2943016960 5761941504 abd_chunk
> $
>
> And total USED/FREE numbers is very strange for me:
>
> $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}'
> 5717965420 9328951088
> $
>
> So, only ~5.7G is used and 9.3G is free! But why this memory is not
> used by ARC anymore and why is it wired and not free?
Could you show the output of "vmstat -s" when in this state?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
Hello,
20.11.2018 15:42, Lev Serebryakov пишет:
I have server which is mostly torrent box. It uses ZFS and equipped
with 16GiB of physical memory. It is running 11-STABLE (r339914 now).
I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago.
I was used to see 13-14GiB of memory in ZFS ARC and it was Ok.
Sometimes it "locks" under heavy disk load due to ARC memory pressure,
but it was bearable, and as ZFS is main reason this server exists, I
didn't limit ARC.
But new revision (r339914) shows very strange behaivor: ARC is no more
than 4GiB, but kernel has 15GiB wired:
Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free
ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other
2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio
It is typical numbers for last week: 15G wired, 237M Free, but only
4252M ARC!
Where is other 11G of memory?!
[...]
And total USED/FREE numbers is very strange for me:
$ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}'
5717965420 9328951088
$
So, only ~5.7G is used and 9.3G is free! But why this memory is not
used by ARC anymore and why is it wired and not free?
I'm getting pretty much same story on recent 11-STABLE from 9th
November. Previous versions didn't have that much questions about memory
usage (and I run several 11-STABLEs).
Eugene.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
