Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-11 Thread John Baldwin
On Wednesday, November 05, 2014 4:52:50 am Andriy Gapon wrote: On 04/11/2014 14:55, Steven Hartland wrote: This is likely spikes in uma zones used by ARC. The VM doesn't ever clean uma zones unless it hits a low memory condition, which explains why your little script helps. Check

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread Steven Hartland
On 05/11/2014 06:15, Marcus Reid wrote: On Tue, Nov 04, 2014 at 06:13:44PM +, Steven Hartland wrote: On 04/11/2014 17:22, Allan Jude wrote: snip... Justin Gibbs and I were helping George from Voxer look at the same issue they are having. They had ~169GB in inact, and only ~60GB being used

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread Andriy Gapon
On 04/11/2014 14:55, Steven Hartland wrote: This is likely spikes in uma zones used by ARC. The VM doesn't ever clean uma zones unless it hits a low memory condition, which explains why your little script helps. Check the output of vmstat -z to confirm. Steve, this is nonsense :-) You

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread Andriy Gapon
On 04/11/2014 14:55, Steven Hartland wrote: This is likely spikes in uma zones used by ARC. The VM doesn't ever clean uma zones unless it hits a low memory condition, which explains why your little script helps. Check the output of vmstat -z to confirm. Steve, this is nonsense :-) You

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread Steven Hartland
On 05/11/2014 09:52, Andriy Gapon wrote: On 04/11/2014 14:55, Steven Hartland wrote: This is likely spikes in uma zones used by ARC. The VM doesn't ever clean uma zones unless it hits a low memory condition, which explains why your little script helps. Check the output of vmstat -z to

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread Dmitriy Makarov
Steven Hartland wrote On 05/11/2014 06:15, Marcus Reid wrote: On Tue, Nov 04, 2014 at 06:13:44PM +, Steven Hartland wrote: On 04/11/2014 17:22, Allan Jude wrote: snip... Justin Gibbs and I were helping George from Voxer look at the same issue they are having. They had ~169GB in inact,

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread James R. Van Artsdalen
On 11/4/2014 5:47 AM, Dmitriy Makarov wrote: Funny thing is that when we manually allocate and release memory, using simple python script: ... Current workaround is to periodically invoke this python script by cron. I wonder if this is related to PR

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread Andriy Gapon
On 05/11/2014 14:36, James R. Van Artsdalen wrote: I wonder if this is related to PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194513 This is against zfs recv and hanging in process state kmem arena but also has a workaround of allocating lots of memory in userland. If something

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread Andriy Gapon
On 05/11/2014 18:32, James R. Van Artsdalen wrote: On 11/5/2014 6:41 AM, Andriy Gapon wrote: If something hangs (appears to hang) and it's ZFS related, then https://wiki.freebsd.org/AvgZfsDeadlockDebug I don't think thezpool history hang is in ZFS or storage layer code: it seems be

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-05 Thread James R. Van Artsdalen
On 11/5/2014 6:41 AM, Andriy Gapon wrote: If something hangs (appears to hang) and it's ZFS related, then https://wiki.freebsd.org/AvgZfsDeadlockDebug I don't think thezpool history hang is in ZFS or storage layer code: it seems be stalled in kernel malloc(). See PID 12105 (zpool history)

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-04 Thread Steven Hartland
This is likely spikes in uma zones used by ARC. The VM doesn't ever clean uma zones unless it hits a low memory condition, which explains why your little script helps. Check the output of vmstat -z to confirm. On 04/11/2014 11:47, Dmitriy Makarov wrote: Hi Current, It seems like there is

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-04 Thread Dmitriy Makarov
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 210, 10, 216, 0, 0 UMA Zones: 2176, 0, 210, 0, 216, 0, 0 UMA Slabs: 80, 0, 2921231, 1024519,133906002, 0, 0 UMA

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-04 Thread Allan Jude
On 11/04/2014 08:22, Dmitriy Makarov wrote: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 210, 10, 216, 0, 0 UMA Zones: 2176, 0, 210, 0, 216, 0, 0 UMA Slabs: 80,

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-04 Thread Ben Perrault
On Nov 4, 2014, at 9:22 AM, Allan Jude allanj...@freebsd.org wrote: On 11/04/2014 08:22, Dmitriy Makarov wrote: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 210, 10, 216, 0, 0 UMA Zones: 2176,

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-04 Thread Steven Hartland
On 04/11/2014 17:22, Allan Jude wrote: snip... Justin Gibbs and I were helping George from Voxer look at the same issue they are having. They had ~169GB in inact, and only ~60GB being used for ARC. Are there any further debugging steps we can recommend to him to help investigate this? The

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-04 Thread Steven Hartland
On 04/11/2014 17:57, Ben Perrault wrote: snip... I would also be interested in any additional debugging steps and would be willing to help test in any way I can - as I've seen the behavior a few times as well. As recently a Sunday evening, I caught a system running with ~44GB ARC but ~117GB

Re: r273165. ZFS ARC: possible memory leak to Inact

2014-11-04 Thread Marcus Reid
On Tue, Nov 04, 2014 at 06:13:44PM +, Steven Hartland wrote: On 04/11/2014 17:22, Allan Jude wrote: snip... Justin Gibbs and I were helping George from Voxer look at the same issue they are having. They had ~169GB in inact, and only ~60GB being used for ARC. Are there any