Re: [zfs-discuss] ARC de-allocation with large ram

2012-10-22 Thread Richard Elling
On Oct 22, 2012, at 6:52 AM, Chris Nagele  wrote:

>> If after it decreases in size it stays there it might be similar to:
>> 
>>7111576 arc shrinks in the absence of memory pressure
> 
> After it dropped, it did build back up. Today is the first day that
> these servers are working under real production load and it is looking
> much better. arcstat is showing some nice numbers for arc, but l2 is
> still building.
> 
> read  hits  miss  hit%  l2read  l2hits  l2miss  l2hit%  arcsz  l2size
> 19K   17K  2.5K872.5K 4902.0K  19   148G371G
> 41K   39K  2.3K942.3K 1842.1K   7   148G371G
> 34K   34K   69498 694  17 677   2   148G371G
> 16K   15K  1.0K931.0K  161.0K   1   148G371G
> 39K   36K  2.3K942.3K  202.3K   0   148G371G
> 23K   22K   74696 746  76 670  10   148G371G
> 49K   47K  1.7K961.7K 2491.5K  14   148G371G
> 23K   21K  1.4K931.4K  381.4K   2   148G371G
> 
> My only guess is that the large zfs send / recv streams were affecting
> the cache when they started and finished.

There are other cases where data is evicted from the ARC, though I don't
have a complete list at my fingertips. For example, if a zvol is closed, then
the data for the zvol is evicted.
 -- richard

> 
> Thanks for the responses and help.
> 
> Chris
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

--

richard.ell...@richardelling.com
+1-760-896-4422









___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ARC de-allocation with large ram

2012-10-22 Thread Chris Nagele
> If after it decreases in size it stays there it might be similar to:
>
> 7111576 arc shrinks in the absence of memory pressure

After it dropped, it did build back up. Today is the first day that
these servers are working under real production load and it is looking
much better. arcstat is showing some nice numbers for arc, but l2 is
still building.

read  hits  miss  hit%  l2read  l2hits  l2miss  l2hit%  arcsz  l2size
 19K   17K  2.5K872.5K 4902.0K  19   148G371G
 41K   39K  2.3K942.3K 1842.1K   7   148G371G
 34K   34K   69498 694  17 677   2   148G371G
 16K   15K  1.0K931.0K  161.0K   1   148G371G
 39K   36K  2.3K942.3K  202.3K   0   148G371G
 23K   22K   74696 746  76 670  10   148G371G
 49K   47K  1.7K961.7K 2491.5K  14   148G371G
 23K   21K  1.4K931.4K  381.4K   2   148G371G

My only guess is that the large zfs send / recv streams were affecting
the cache when they started and finished.

Thanks for the responses and help.

Chris
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ARC de-allocation with large ram

2012-10-22 Thread Tomas Forsman
On 22 October, 2012 - Robert Milkowski sent me these 3,6K bytes:

> Hi,
> 
> If after it decreases in size it stays there it might be similar to:
> 
>   7111576 arc shrinks in the absence of memory pressure
> 
> Also, see document:
> 
>   ZFS ARC can shrink down without memory pressure result in slow
> performance [ID 1404581.1]
> 
> Specifically, check if arc_no_grow is set to 1 after the cache size is
> decreased, and if it stays that way.
> 
> The fix is in one of the SRUs and I think it should be in 11.1
> I don't know if it was fixed in Illumos or even if Illumos was affected by
> this at all.

The code that affects bug 7111576 was introduced between s10 and s11.

/Tomas
-- 
Tomas Forsman, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of UmeƄ
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ARC de-allocation with large ram

2012-10-22 Thread Robert Milkowski
Hi,

If after it decreases in size it stays there it might be similar to:

7111576 arc shrinks in the absence of memory pressure

Also, see document:

ZFS ARC can shrink down without memory pressure result in slow
performance [ID 1404581.1]

Specifically, check if arc_no_grow is set to 1 after the cache size is
decreased, and if it stays that way.

The fix is in one of the SRUs and I think it should be in 11.1
I don't know if it was fixed in Illumos or even if Illumos was affected by
this at all.


-- 
Robert Milkowski
http://milek.blogspot.com


> -Original Message-
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Chris Nagele
> Sent: 20 October 2012 18:47
> To: zfs-discuss@opensolaris.org
> Subject: [zfs-discuss] ARC de-allocation with large ram
> 
> Hi. We're running OmniOS as a ZFS storage server. For some reason, our
> arc cache will grow to a certain point, then suddenly drops. I used
> arcstat to catch it in action, but I was not able to capture what else
> was going on in the system at the time. I'll do that next.
> 
> read  hits  miss  hit%  l2read  l2hits  l2miss  l2hit%  arcsz  l2size
>  166   166 0   100   0   0   0   085G225G
> 5.9K  5.9K 0   100   0   0   0   085G225G
>  755   7154094  40   0  40   084G225G
>  17K   17K 0   100   0   0   0   067G225G
>  409   3951496  14   0  14   049G225G
>  388   3642493  24   0  24   041G225G
>  37K   37K2099  20   6  14  3040G225G
> 
> For reference, it's a 12TB pool with 512GB SSD L2 ARC and 198GB RAM.
> We have nothing else running on the system except NFS. We are also not
> using dedupe. Here is the output of memstat at one point:
> 
> # echo ::memstat | mdb -k
> Page SummaryPagesMB  %Tot
>      
> Kernel   19061902 74460   38%
> ZFS File Data28237282110301   56%
> Anon43112   1680%
> Exec and libs1522 50%
> Page cache  13509520%
> Free (cachelist) 6366240%
> Free (freelist)   2958527 115566%
> 
> Total5030196571
> Physical 50322219196571
> 
> According to "prstat -s rss" nothing else is consuming the memory.
> 
>592 root   33M   26M sleep   590   0:00:33 0.0% fmd/27
> 12 root   13M   11M sleep   590   0:00:08 0.0%
> svc.configd/21
>641 root   12M   11M sleep   590   0:04:48 0.0% snmpd/1
> 10 root   14M   10M sleep   590   0:00:03 0.0%
> svc.startd/16
>342 root   12M 9084K sleep   590   0:00:15 0.0% hald/5
>321 root   14M 8652K sleep   590   0:03:00 0.0% nscd/52
> 
> So far I can't figure out what could be causing this. The only other
> thing I can think of is that we have a bunch of zfs send/receive
> operations going on as backups across 10 datasets in the pool. I  am
> not sure how snapshots and send/receive affect the arc. Does anyone
> else have any ideas?
> 
> Thanks,
> Chris
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss