Re: r273165. ZFS ARC: possible memory leak to Inact
On Wednesday, November 05, 2014 4:52:50 am Andriy Gapon wrote: > On 04/11/2014 14:55, Steven Hartland wrote: > > This is likely spikes in uma zones used by ARC. > > > > The VM doesn't ever clean uma zones unless it hits a low memory condition, which > > explains why your little script helps. > > > > Check the output of vmstat -z to confirm. > > Steve, > > this is nonsense :-) You know perfectly well that UMA memory is Wired not Inactive. Grab the code at www.freebsd.org/~jhb/vm_objects/. Build and load the kld and then use the vm_objects binary to generate a list of the active VM objects in the system along with counts of how many active / inactive pages each object contains. For your case with lots of inactive memory, you probably want something like 'vm_objects | sort -n -k 2'. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 11/5/2014 6:41 AM, Andriy Gapon wrote: > If something hangs (appears to hang) and it's ZFS related, then > https://wiki.freebsd.org/AvgZfsDeadlockDebug > I don't think the"zpool history" hang is in ZFS or storage layer code: it seems be stalled in kernel malloc(). See PID 12105 (zpool history) below. SUPERTEX:/root# uname -a FreeBSD SUPERTEX.housenet.jrv 10.1-PRERELEASE FreeBSD 10.1-PRERELEASE #3 r273476M: Wed Oct 22 15:05:29 CDT 2014 r...@supertex.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 SUPERTEX:/root# procstat -kk -a > kk SUPERTEX:/root# grep zio_wait kk SUPERTEX:/root# grep zio_done kk SUPERTEX:/root# grep zio_int kk SUPERTEX:/root# grep zfs_ kk 0 100475 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe 0 101018 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe 0 101522 kernel zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe 12105 101599 zpool-mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d kmem_malloc+0x33 uma_large_malloc+0x49 malloc+0x43 zfs_ioc_pool_get_history+0xbd zfsdev_ioctl+0x5ca devfs_ioctl_f+0x139 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb SUPERTEX:/root# grep dmu_ kk SUPERTEX:/root# grep arc_ kk 3 100125 zfskern arc_reclaim_thre mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x18b arc_reclaim_thread+0x288 fork_exit+0x9a fork_trampoline+0xe 3 100126 zfskern l2arc_feed_threa mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x18b l2arc_feed_thread+0x19f fork_exit+0x9a fork_trampoline+0xe SUPERTEX:/root# grep buf_ kk 20 100139 bufdaemon-mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e buf_daemon+0x78 fork_exit+0x9a fork_trampoline+0xe SUPERTEX:/root# ps lp 12105 UID PID PPID CPU PRI NIVSZ RSS MWCHAN STAT TT TIME COMMAND 0 12105 12104 0 20 0 105692 35984 kmem are D - 0:03.76 zpool history BI SUPERTEX:/root# ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 05/11/2014 18:32, James R. Van Artsdalen wrote: > On 11/5/2014 6:41 AM, Andriy Gapon wrote: >> If something hangs (appears to hang) and it's ZFS related, then >> https://wiki.freebsd.org/AvgZfsDeadlockDebug >> > > I don't think the"zpool history" hang is in ZFS or storage layer code: > it seems be stalled in kernel malloc(). See PID 12105 (zpool > history) below. The wiki page has this underscored line :-) when reporting a problem please always include full information about thread stacks, don't cherry pick; the output can be large, upload it somewhere and post a link > SUPERTEX:/root# uname -a > FreeBSD SUPERTEX.housenet.jrv 10.1-PRERELEASE FreeBSD 10.1-PRERELEASE #3 > r273476M: Wed Oct 22 15:05:29 CDT 2014 > r...@supertex.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 > SUPERTEX:/root# procstat -kk -a > kk > SUPERTEX:/root# grep zio_wait kk > SUPERTEX:/root# grep zio_done kk > SUPERTEX:/root# grep zio_int kk > SUPERTEX:/root# grep zfs_ kk > 0 100475 kernel zfs_vn_rele_task mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a > fork_trampoline+0xe > 0 101018 kernel zfs_vn_rele_task mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a > fork_trampoline+0xe > 0 101522 kernel zfs_vn_rele_task mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a > fork_trampoline+0xe > 12105 101599 zpool-mi_switch+0xe1 > sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d > kmem_malloc+0x33 uma_large_malloc+0x49 malloc+0x43 > zfs_ioc_pool_get_history+0xbd zfsdev_ioctl+0x5ca devfs_ioctl_f+0x139 > kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb > SUPERTEX:/root# grep dmu_ kk > SUPERTEX:/root# grep arc_ kk > 3 100125 zfskern arc_reclaim_thre mi_switch+0xe1 > sleepq_timedwait+0x3a _cv_timedwait_sbt+0x18b arc_reclaim_thread+0x288 > fork_exit+0x9a fork_trampoline+0xe > 3 100126 zfskern l2arc_feed_threa mi_switch+0xe1 > sleepq_timedwait+0x3a _cv_timedwait_sbt+0x18b l2arc_feed_thread+0x19f > fork_exit+0x9a fork_trampoline+0xe > SUPERTEX:/root# grep buf_ kk >20 100139 bufdaemon-mi_switch+0xe1 > sleepq_timedwait+0x3a _sleep+0x26e buf_daemon+0x78 fork_exit+0x9a > fork_trampoline+0xe > SUPERTEX:/root# ps lp 12105 > UID PID PPID CPU PRI NIVSZ RSS MWCHAN STAT TT TIME COMMAND > 0 12105 12104 0 20 0 105692 35984 kmem are D - 0:03.76 zpool > history BI > SUPERTEX:/root# > > -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 05/11/2014 14:36, James R. Van Artsdalen wrote: > I wonder if this is related to PR > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194513 > > This is against "zfs recv" and hanging in process state "kmem arena" & > but also has a workaround of allocating lots of memory in userland. If something hangs (appears to hang) and it's ZFS related, then https://wiki.freebsd.org/AvgZfsDeadlockDebug > But I do not see a lot of inactive with that PR. > > "zpool history" also hangs sometimes in "kmem arena" but I do not have > a workaround for that. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 11/4/2014 5:47 AM, Dmitriy Makarov wrote: > Funny thing is that when we manually allocate and release memory, using > simple python script: ... > > Current workaround is to periodically invoke this python script by cron. > I wonder if this is related to PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194513 This is against "zfs recv" and hanging in process state "kmem arena" & but also has a workaround of allocating lots of memory in userland. But I do not see a lot of inactive with that PR. "zpool history" also hangs sometimes in "kmem arena" but I do not have a workaround for that. This PR is filed against 10-STABLE but confirmed against CURRENT too. SUPERTEX:/root# uname -a FreeBSD SUPERTEX.housenet.jrv 10.1-PRERELEASE FreeBSD 10.1-PRERELEASE #3 r273476M: Wed Oct 22 15:05:29 CDT 2014 r...@supertex.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 SUPERTEX:/root# top last pid: 37286; load averages: 0.03, 0.05, 0.05 up 11+11:24:34 06:25:46 39 processes: 1 running, 38 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 6444K Active, 57M Inact, 6475M Wired, 25G Free ARC: 4753M Total, 862M MFU, 2765M MRU, 52K Anon, 139M Header, 986M Other Swap: 31G Total, 21M Used, 31G Free PID USERNAMETHR PRI NICE SIZERES STATE C TIMEWCPU COMMAND 676 root 1 200 25456K 1048K select 8 0:22 0.00% ntpd 723 root 1 200 24112K 1472K select 13 0:09 0.00% sendmail 12105 root 1 200 103M 35984K kmem a 11 0:04 0.00% zpool 693 root 1 200 30676K 1684K nanslp 10 0:03 0.00% smartd 519 root 1 200 14508K 684K select 5 0:02 0.00% syslogd ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
Steven Hartland wrote > On 05/11/2014 06:15, Marcus Reid wrote: >> On Tue, Nov 04, 2014 at 06:13:44PM +, Steven Hartland wrote: >>> On 04/11/2014 17:22, Allan Jude wrote: >>>> snip... >>>> Justin Gibbs and I were helping George from Voxer look at the same >>>> issue >>>> they are having. They had ~169GB in inact, and only ~60GB being used >>>> for >>>> ARC. >>>> >>>> Are there any further debugging steps we can recommend to him to help >>>> investigate this? >>> The various scripts attached to the ZS ARC behavior problem and fix PR >>> will help provide detail this. >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 >>> >>> I've seen it here where there's been bursts of ZFS I/O specifically >>> write bursts. >>> >>> What happens is that ZFS will consume large amounts of space in various >>> UMA zones to accommodate these bursts. >> If you push the vmstat -z that he provided through the arc summary >> script, you'll see that this is not what is happening. His uma stats >> match up with his arc, and do not account for his inactive memory. >> >> uma script summary: >> >> Totals >> oused: 5.860GB, ofree: 1.547GB, ototal: 7.407GB >> zused: 56.166GB, zfree: 3.918GB, ztotal: 60.084GB >> used: 62.026GB, free: 5.465GB, total: 67.491GB >> >> His provided top stats: >> >> Mem: 19G Active, 20G Inact, 81G Wired, 59M Cache, 3308M Buf, 4918M >> Free >> ARC: 66G Total, 6926M MFU, 54G MRU, 8069K Anon, 899M Header, 5129M >> Other >> >> >> The big uma buckets (zio_buf_16384 and zio_data_buf_131072, 18.002GB and >> 28.802GB respectively) are nearly 0% free. >> > Still potentially accounts for 5.4GB of your 20GB inact. > > The rest could be malloc backed allocations? No. There are few reasons for that. The first one is that Inact constantly grows, and 20GiB you see were 50GiBs before we ran the script. (We have to run it periodically or else our production server will grow slower and slower) The second argumens is that our codebase is the same, the only thing that have changed is OS version. In the previous version Inact was dramatically much smaller: ~hundrets of megabytes. -- View this message in context: http://freebsd.1045724.n5.nabble.com/r273165-ZFS-ARC-possible-memory-leak-to-Inact-tp5962410p5962711.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 05/11/2014 09:52, Andriy Gapon wrote: On 04/11/2014 14:55, Steven Hartland wrote: This is likely spikes in uma zones used by ARC. The VM doesn't ever clean uma zones unless it hits a low memory condition, which explains why your little script helps. Check the output of vmstat -z to confirm. Steve, this is nonsense :-) You know perfectly well that UMA memory is Wired not Inactive. I'll wake up in a bit honest, thanks for the slap ;-) ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 04/11/2014 14:55, Steven Hartland wrote: > This is likely spikes in uma zones used by ARC. > > The VM doesn't ever clean uma zones unless it hits a low memory condition, > which > explains why your little script helps. > > Check the output of vmstat -z to confirm. Steve, this is nonsense :-) You know perfectly well that UMA memory is Wired not Inactive. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 04/11/2014 14:55, Steven Hartland wrote: > This is likely spikes in uma zones used by ARC. > > The VM doesn't ever clean uma zones unless it hits a low memory condition, > which > explains why your little script helps. > > Check the output of vmstat -z to confirm. Steve, this is nonsense :-) You know perfectly well that UMA memory is Wired not Inactive. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 05/11/2014 06:15, Marcus Reid wrote: On Tue, Nov 04, 2014 at 06:13:44PM +, Steven Hartland wrote: On 04/11/2014 17:22, Allan Jude wrote: snip... Justin Gibbs and I were helping George from Voxer look at the same issue they are having. They had ~169GB in inact, and only ~60GB being used for ARC. Are there any further debugging steps we can recommend to him to help investigate this? The various scripts attached to the ZS ARC behavior problem and fix PR will help provide detail this. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 I've seen it here where there's been bursts of ZFS I/O specifically write bursts. What happens is that ZFS will consume large amounts of space in various UMA zones to accommodate these bursts. If you push the vmstat -z that he provided through the arc summary script, you'll see that this is not what is happening. His uma stats match up with his arc, and do not account for his inactive memory. uma script summary: Totals oused: 5.860GB, ofree: 1.547GB, ototal: 7.407GB zused: 56.166GB, zfree: 3.918GB, ztotal: 60.084GB used: 62.026GB, free: 5.465GB, total: 67.491GB His provided top stats: Mem: 19G Active, 20G Inact, 81G Wired, 59M Cache, 3308M Buf, 4918M Free ARC: 66G Total, 6926M MFU, 54G MRU, 8069K Anon, 899M Header, 5129M Other The big uma buckets (zio_buf_16384 and zio_data_buf_131072, 18.002GB and 28.802GB respectively) are nearly 0% free. Still potentially accounts for 5.4GB of your 20GB inact. The rest could be malloc backed allocations? Regards Steve ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On Tue, Nov 04, 2014 at 06:13:44PM +, Steven Hartland wrote: > > On 04/11/2014 17:22, Allan Jude wrote: > > snip... > > Justin Gibbs and I were helping George from Voxer look at the same issue > > they are having. They had ~169GB in inact, and only ~60GB being used for > > ARC. > > > > Are there any further debugging steps we can recommend to him to help > > investigate this? > The various scripts attached to the ZS ARC behavior problem and fix PR > will help provide detail this. > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 > > I've seen it here where there's been bursts of ZFS I/O specifically > write bursts. > > What happens is that ZFS will consume large amounts of space in various > UMA zones to accommodate these bursts. If you push the vmstat -z that he provided through the arc summary script, you'll see that this is not what is happening. His uma stats match up with his arc, and do not account for his inactive memory. uma script summary: Totals oused: 5.860GB, ofree: 1.547GB, ototal: 7.407GB zused: 56.166GB, zfree: 3.918GB, ztotal: 60.084GB used: 62.026GB, free: 5.465GB, total: 67.491GB His provided top stats: Mem: 19G Active, 20G Inact, 81G Wired, 59M Cache, 3308M Buf, 4918M Free ARC: 66G Total, 6926M MFU, 54G MRU, 8069K Anon, 899M Header, 5129M Other The big uma buckets (zio_buf_16384 and zio_data_buf_131072, 18.002GB and 28.802GB respectively) are nearly 0% free. Marcus > The VM only triggers UMA reclaim when it sees pressure, however if the > main memory consumer is ZFS ARC its possible that the require pressure > will not be applied because when allocating ARC ZFS takes into account > free memory. > > The result is it will back off its memory requirements before the > reclaim is triggered leaving all the space allocated but not used. > > I was playing around with a patch, on that bug report, which added clear > down of UMA within ZFS ARC to avoid just this behavior, but its very > much me playing for testing the theory only. > > From what I've seen UMA needs something like the coloring which can be > used to trigger clear down over time to prevent UMA zones sitting their > eating large amounts of memory like they currently do. > > Regards > Steve > ___ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 04/11/2014 17:57, Ben Perrault wrote: snip... I would also be interested in any additional debugging steps and would be willing to help test in any way I can - as I've seen the behavior a few times as well. As recently a Sunday evening, I caught a system running with ~44GB ARC but ~117GB inactive. You should find the UMA summary script quite helpful in this regard: https://bugs.freebsd.org/bugzilla/attachment.cgi?id=147754 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
On 04/11/2014 17:22, Allan Jude wrote: snip... Justin Gibbs and I were helping George from Voxer look at the same issue they are having. They had ~169GB in inact, and only ~60GB being used for ARC. Are there any further debugging steps we can recommend to him to help investigate this? The various scripts attached to the ZS ARC behavior problem and fix PR will help provide detail this. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 I've seen it here where there's been bursts of ZFS I/O specifically write bursts. What happens is that ZFS will consume large amounts of space in various UMA zones to accommodate these bursts. The VM only triggers UMA reclaim when it sees pressure, however if the main memory consumer is ZFS ARC its possible that the require pressure will not be applied because when allocating ARC ZFS takes into account free memory. The result is it will back off its memory requirements before the reclaim is triggered leaving all the space allocated but not used. I was playing around with a patch, on that bug report, which added clear down of UMA within ZFS ARC to avoid just this behavior, but its very much me playing for testing the theory only. From what I've seen UMA needs something like the coloring which can be used to trigger clear down over time to prevent UMA zones sitting their eating large amounts of memory like they currently do. Regards Steve ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
0, 0 >> pf frags:80, 0, 0, 0, 0, 0, 0 >> pf frag entries: 32, 4, 0, 0, 0, 0, 0 >> pf state scrubs: 40, 0, 0, 0, 0, 0, 0 >> KNOTE: 128, 0, 13343,1568,2119230288, 0, 0 >> socket: 728, 4192760, 31124,1581,260689101, 0, 0 >> ipq: 56, 225283, 0, 0, 0, 0, 0 >> udp_inpcb: 400, 4192760, 46, 484,18539506, 0, 0 >> udpcb: 24, 4192869, 46,4296,18539506, 0, 0 >> tcp_inpcb: 400, 4192760, 42550,1050,241905139, 0, 0 >> tcpcb: 1032, 4192761, 14734, 830,241905139, 0, 0 >> tcptw: 80, 27800, 27800, 0,100020089,89206796, >> 0 >> syncache: 168, 15364, 0, 805,137341445, 0, 0 >> hostcache: 136, 15370, 57, 233, 759, 0, 0 >> sackhole:32, 0, 0,3125, 19180, 0, 0 >> sctp_ep: 1400, 4192760, 0, 0, 0, 0, 0 >> sctp_asoc: 2408, 4, 0, 0, 0, 0, 0 >> sctp_laddr: 48, 80012, 0, 0, 3, 0, 0 >> sctp_raddr: 720, 8, 0, 0, 0, 0, 0 >> sctp_chunk: 136, 400026, 0, 0, 0, 0, 0 >> sctp_readq: 104, 400026, 0, 0, 0, 0, 0 >> sctp_stream_msg_out:104, 400026, 0, 0, 0, 0, 0 >> sctp_asconf: 40, 40, 0, 0, 0, 0, 0 >> sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 >> udplite_inpcb: 400, 4192760, 0, 0, 0, 0, 0 >> ripcb: 400, 4192760, 0, 60, 6, 0, 0 >> unpcb: 240, 4192768,1166,1074, 28, 0, 0 >> rtentry:200, 0, 8, 92, 8, 0, 0 >> selfd: 56, 0,2339,3270,6167642044, 0, 0 >> SWAPMETA: 288, 16336788, 0, 0, 0, 0, 0 >> FFS inode: 168, 0,1032,1084, 1308978, 0, 0 >> FFS1 dinode:128, 0, 0, 0, 0, 0, 0 >> FFS2 dinode:256, 0,1032,1098, 1308978, 0, 0 >> NCLNODE:528, 0, 0, 0, 0, 0, 0 >> >> this is staticticts after script helped to reclaim memory. >> >> Here's top statistics: >> >> Mem: 19G Active, 20G Inact, 81G Wired, 59M Cache, 3308M Buf, 4918M Free >> ARC: 66G Total, 6926M MFU, 54G MRU, 8069K Anon, 899M Header, 5129M Other >> >> >> >> Steven Hartland wrote >>> This is likely spikes in uma zones used by ARC. >>> >>> The VM doesn't ever clean uma zones unless it hits a low memory >>> condition, which explains why your little script helps. >>> >>> Check the output of vmstat -z to confirm. >>> >>>> On 04/11/2014 11:47, Dmitriy Makarov wrote: >>>> Hi Current, >>>> >>>> It seems like there is constant flow (leak) of memory from ARC to Inact >>>> in FreeBSD 11.0-CURRENT #0 r273165. >>>> >>>> Normally, our system (FreeBSD 11.0-CURRENT #5 r260625) keeps ARC size >>>> very close to vfs.zfs.arc_max: >>>> >>>> Mem: 16G Active, 324M Inact, 105G Wired, 1612M Cache, 3308M Buf, 1094M >>>> Free >>>> ARC: 88G Total, 2100M MFU, 78G MRU, 39M Anon, 2283M Header, 6162M Other >>>> >>>> >>>> But after an upgrade to (FreeBSD 11.0-CURRENT #0 r273165) we observe >>>> enormous numbers of Inact memory in the top: >>>> >>>> Mem: 21G Active, 45G Inact, 56G Wired, 357M Cache, 3308M Buf, 1654M Free >>>> ARC: 42G Total, 6025M MFU, 30G MRU, 30M Anon, 819M Header, 5214M Other >>>> >>>> Funny thing is that when we manually allocate and release memory, using >>>> simple python script: >>>> >>>> #!/usr/local/bin/python2.7 >>>> >>>> import sys >>>> import time >>>> >>>> if len(sys.argv) != 2: >>>> print "usage: fillmem >>> >>> " >>>> sys.exit() >>>> >>>> count = int(sys.argv[1]) >>>> >>>> megabyte = (0,) * (1024 * 1024
Re: r273165. ZFS ARC: possible memory leak to Inact
tcptw: 80, 27800, 27800, 0,100020089,89206796, > 0 > syncache: 168, 15364, 0, 805,137341445, 0, 0 > hostcache: 136, 15370, 57, 233, 759, 0, 0 > sackhole:32, 0, 0,3125, 19180, 0, 0 > sctp_ep: 1400, 4192760, 0, 0, 0, 0, 0 > sctp_asoc: 2408, 4, 0, 0, 0, 0, 0 > sctp_laddr: 48, 80012, 0, 0, 3, 0, 0 > sctp_raddr: 720, 8, 0, 0, 0, 0, 0 > sctp_chunk: 136, 400026, 0, 0, 0, 0, 0 > sctp_readq: 104, 400026, 0, 0, 0, 0, 0 > sctp_stream_msg_out:104, 400026, 0, 0, 0, 0, 0 > sctp_asconf: 40, 40, 0, 0, 0, 0, 0 > sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 > udplite_inpcb: 400, 4192760, 0, 0, 0, 0, 0 > ripcb: 400, 4192760, 0, 60, 6, 0, 0 > unpcb: 240, 4192768,1166,1074, 28, 0, 0 > rtentry:200, 0, 8, 92, 8, 0, 0 > selfd: 56, 0,2339,3270,6167642044, 0, 0 > SWAPMETA: 288, 16336788, 0, 0, 0, 0, 0 > FFS inode: 168, 0,1032,1084, 1308978, 0, 0 > FFS1 dinode:128, 0, 0, 0, 0, 0, 0 > FFS2 dinode:256, 0,1032,1098, 1308978, 0, 0 > NCLNODE:528, 0, 0, 0, 0, 0, 0 > > this is staticticts after script helped to reclaim memory. > > Here's top statistics: > > Mem: 19G Active, 20G Inact, 81G Wired, 59M Cache, 3308M Buf, 4918M Free > ARC: 66G Total, 6926M MFU, 54G MRU, 8069K Anon, 899M Header, 5129M Other > > > > Steven Hartland wrote >> This is likely spikes in uma zones used by ARC. >> >> The VM doesn't ever clean uma zones unless it hits a low memory >> condition, which explains why your little script helps. >> >> Check the output of vmstat -z to confirm. >> >> On 04/11/2014 11:47, Dmitriy Makarov wrote: >>> Hi Current, >>> >>> It seems like there is constant flow (leak) of memory from ARC to Inact >>> in FreeBSD 11.0-CURRENT #0 r273165. >>> >>> Normally, our system (FreeBSD 11.0-CURRENT #5 r260625) keeps ARC size >>> very close to vfs.zfs.arc_max: >>> >>> Mem: 16G Active, 324M Inact, 105G Wired, 1612M Cache, 3308M Buf, 1094M >>> Free >>> ARC: 88G Total, 2100M MFU, 78G MRU, 39M Anon, 2283M Header, 6162M Other >>> >>> >>> But after an upgrade to (FreeBSD 11.0-CURRENT #0 r273165) we observe >>> enormous numbers of Inact memory in the top: >>> >>> Mem: 21G Active, 45G Inact, 56G Wired, 357M Cache, 3308M Buf, 1654M Free >>> ARC: 42G Total, 6025M MFU, 30G MRU, 30M Anon, 819M Header, 5214M Other >>> >>> Funny thing is that when we manually allocate and release memory, using >>> simple python script: >>> >>> #!/usr/local/bin/python2.7 >>> >>> import sys >>> import time >>> >>> if len(sys.argv) != 2: >>> print "usage: fillmem >> >> " >>> sys.exit() >>> >>> count = int(sys.argv[1]) >>> >>> megabyte = (0,) * (1024 * 1024 / 8) >>> >>> data = megabyte * count >>> >>> as: >>> >>> # ./simple_script 1 >>> >>> all those allocated megabyes 'migrate' from Inact to Free, and afterwards >>> they are 'eaten' by ARC with no problem. >>> Until Inact slowly grows back to the number it was before we ran the >>> script. >>> >>> Current workaround is to periodically invoke this python script by cron. >>> This is an ugly workaround and we really don't like it on our production >>> >>> >>> To answer possible questions about ARC efficience: >>> Cache efficiency drops dramatically with every GiB pushed off the ARC. >>> >>> Before upgrade: >>> Cache Hit Ratio:99.38% >>> >>> After upgrade: >>> Cache Hit Ratio:81.95% >>> >>> We believe that ARC misbehaves and we ask your assistance. >>> >>> >>> -- >>> >>> Some values from conf
Re: r273165. ZFS ARC: possible memory leak to Inact
ripcb: 400, 4192760, 0, 60, 6, 0, 0 unpcb: 240, 4192768,1166,1074, 28, 0, 0 rtentry:200, 0, 8, 92, 8, 0, 0 selfd: 56, 0,2339,3270,6167642044, 0, 0 SWAPMETA: 288, 16336788, 0, 0, 0, 0, 0 FFS inode: 168, 0,1032,1084, 1308978, 0, 0 FFS1 dinode:128, 0, 0, 0, 0, 0, 0 FFS2 dinode:256, 0,1032,1098, 1308978, 0, 0 NCLNODE:528, 0, 0, 0, 0, 0, 0 this is staticticts after script helped to reclaim memory. Here's top statistics: Mem: 19G Active, 20G Inact, 81G Wired, 59M Cache, 3308M Buf, 4918M Free ARC: 66G Total, 6926M MFU, 54G MRU, 8069K Anon, 899M Header, 5129M Other Steven Hartland wrote > This is likely spikes in uma zones used by ARC. > > The VM doesn't ever clean uma zones unless it hits a low memory > condition, which explains why your little script helps. > > Check the output of vmstat -z to confirm. > > On 04/11/2014 11:47, Dmitriy Makarov wrote: >> Hi Current, >> >> It seems like there is constant flow (leak) of memory from ARC to Inact >> in FreeBSD 11.0-CURRENT #0 r273165. >> >> Normally, our system (FreeBSD 11.0-CURRENT #5 r260625) keeps ARC size >> very close to vfs.zfs.arc_max: >> >> Mem: 16G Active, 324M Inact, 105G Wired, 1612M Cache, 3308M Buf, 1094M >> Free >> ARC: 88G Total, 2100M MFU, 78G MRU, 39M Anon, 2283M Header, 6162M Other >> >> >> But after an upgrade to (FreeBSD 11.0-CURRENT #0 r273165) we observe >> enormous numbers of Inact memory in the top: >> >> Mem: 21G Active, 45G Inact, 56G Wired, 357M Cache, 3308M Buf, 1654M Free >> ARC: 42G Total, 6025M MFU, 30G MRU, 30M Anon, 819M Header, 5214M Other >> >> Funny thing is that when we manually allocate and release memory, using >> simple python script: >> >> #!/usr/local/bin/python2.7 >> >> import sys >> import time >> >> if len(sys.argv) != 2: >> print "usage: fillmem > > " >> sys.exit() >> >> count = int(sys.argv[1]) >> >> megabyte = (0,) * (1024 * 1024 / 8) >> >> data = megabyte * count >> >> as: >> >> # ./simple_script 1 >> >> all those allocated megabyes 'migrate' from Inact to Free, and afterwards >> they are 'eaten' by ARC with no problem. >> Until Inact slowly grows back to the number it was before we ran the >> script. >> >> Current workaround is to periodically invoke this python script by cron. >> This is an ugly workaround and we really don't like it on our production >> >> >> To answer possible questions about ARC efficience: >> Cache efficiency drops dramatically with every GiB pushed off the ARC. >> >> Before upgrade: >> Cache Hit Ratio:99.38% >> >> After upgrade: >> Cache Hit Ratio:81.95% >> >> We believe that ARC misbehaves and we ask your assistance. >> >> >> -- >> >> Some values from configs. >> >> HW: 128GB RAM, LSI HBA controller with 36 disks (stripe of mirrors). >> >> top output: >> >> In /boot/loader.conf : >> vm.kmem_size="110G" >> vfs.zfs.arc_max="90G" >> vfs.zfs.arc_min="42G" >> vfs.zfs.txg.timeout="10" >> >> ------- >> >> Thanks. >> >> Regards, >> Dmitriy >> ___ >> > freebsd-current@ > mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to " > freebsd-current-unsubscribe@ > " > > ___ > freebsd-current@ > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@ > " -- View this message in context: http://freebsd.1045724.n5.nabble.com/r273165-ZFS-ARC-possible-memory-leak-to-Inact-tp5962410p5962421.html Sent from the freebsd-current mailing list archive at Nabble.com. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: r273165. ZFS ARC: possible memory leak to Inact
This is likely spikes in uma zones used by ARC. The VM doesn't ever clean uma zones unless it hits a low memory condition, which explains why your little script helps. Check the output of vmstat -z to confirm. On 04/11/2014 11:47, Dmitriy Makarov wrote: Hi Current, It seems like there is constant flow (leak) of memory from ARC to Inact in FreeBSD 11.0-CURRENT #0 r273165. Normally, our system (FreeBSD 11.0-CURRENT #5 r260625) keeps ARC size very close to vfs.zfs.arc_max: Mem: 16G Active, 324M Inact, 105G Wired, 1612M Cache, 3308M Buf, 1094M Free ARC: 88G Total, 2100M MFU, 78G MRU, 39M Anon, 2283M Header, 6162M Other But after an upgrade to (FreeBSD 11.0-CURRENT #0 r273165) we observe enormous numbers of Inact memory in the top: Mem: 21G Active, 45G Inact, 56G Wired, 357M Cache, 3308M Buf, 1654M Free ARC: 42G Total, 6025M MFU, 30G MRU, 30M Anon, 819M Header, 5214M Other Funny thing is that when we manually allocate and release memory, using simple python script: #!/usr/local/bin/python2.7 import sys import time if len(sys.argv) != 2: print "usage: fillmem " sys.exit() count = int(sys.argv[1]) megabyte = (0,) * (1024 * 1024 / 8) data = megabyte * count as: # ./simple_script 1 all those allocated megabyes 'migrate' from Inact to Free, and afterwards they are 'eaten' by ARC with no problem. Until Inact slowly grows back to the number it was before we ran the script. Current workaround is to periodically invoke this python script by cron. This is an ugly workaround and we really don't like it on our production To answer possible questions about ARC efficience: Cache efficiency drops dramatically with every GiB pushed off the ARC. Before upgrade: Cache Hit Ratio:99.38% After upgrade: Cache Hit Ratio:81.95% We believe that ARC misbehaves and we ask your assistance. -- Some values from configs. HW: 128GB RAM, LSI HBA controller with 36 disks (stripe of mirrors). top output: In /boot/loader.conf : vm.kmem_size="110G" vfs.zfs.arc_max="90G" vfs.zfs.arc_min="42G" vfs.zfs.txg.timeout="10" --- Thanks. Regards, Dmitriy ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
r273165. ZFS ARC: possible memory leak to Inact
Hi Current, It seems like there is constant flow (leak) of memory from ARC to Inact in FreeBSD 11.0-CURRENT #0 r273165. Normally, our system (FreeBSD 11.0-CURRENT #5 r260625) keeps ARC size very close to vfs.zfs.arc_max: Mem: 16G Active, 324M Inact, 105G Wired, 1612M Cache, 3308M Buf, 1094M Free ARC: 88G Total, 2100M MFU, 78G MRU, 39M Anon, 2283M Header, 6162M Other But after an upgrade to (FreeBSD 11.0-CURRENT #0 r273165) we observe enormous numbers of Inact memory in the top: Mem: 21G Active, 45G Inact, 56G Wired, 357M Cache, 3308M Buf, 1654M Free ARC: 42G Total, 6025M MFU, 30G MRU, 30M Anon, 819M Header, 5214M Other Funny thing is that when we manually allocate and release memory, using simple python script: #!/usr/local/bin/python2.7 import sys import time if len(sys.argv) != 2: print "usage: fillmem " sys.exit() count = int(sys.argv[1]) megabyte = (0,) * (1024 * 1024 / 8) data = megabyte * count as: # ./simple_script 1 all those allocated megabyes 'migrate' from Inact to Free, and afterwards they are 'eaten' by ARC with no problem. Until Inact slowly grows back to the number it was before we ran the script. Current workaround is to periodically invoke this python script by cron. This is an ugly workaround and we really don't like it on our production To answer possible questions about ARC efficience: Cache efficiency drops dramatically with every GiB pushed off the ARC. Before upgrade: Cache Hit Ratio:99.38% After upgrade: Cache Hit Ratio:81.95% We believe that ARC misbehaves and we ask your assistance. -- Some values from configs. HW: 128GB RAM, LSI HBA controller with 36 disks (stripe of mirrors). top output: In /boot/loader.conf : vm.kmem_size="110G" vfs.zfs.arc_max="90G" vfs.zfs.arc_min="42G" vfs.zfs.txg.timeout="10" --- Thanks. Regards, Dmitriy ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
