On Apr 8, 2014, at 12:22 PM, Rajesh Poddar <[email protected]> wrote:
> I have a system with 64gb of ram that I'm running smartos on. It has 5
> Windows Server 2012 KVM VMs each with 8GB of memory. There is also a smartos
> zone with 32GB max_physical_memory that runs a very lightweight file server
> that i wrote in Go. The file server typically consumes about 3-4GB of memory.
> Every so often (like once in 10 minutes or so) the fileserver massively slows
> down. I noticed that normally the RSS entry for the fileserver in the output
> of prstat stays stable. However, these periods of slowdown perfectly coincide
> with the RSS values decreasing. This led me to conclude that memory allocated
> to the fileserver is being paged out. Furthermore the fraction of misses as
> shown in the output of arcstat also masively increases during these periods.
The hit/miss ratio or fractions are not very interesting. The magnitude is. 50%
hits for 2 accesses/sec
is very different than 50% hit for 200,000 accesses/sec. A more meaningful way
to describe this is:
150k accesses/sec with 47% hit rate, or better yet:
demand
data
hits (dhit)
misses (dmis)
metadata
hits (mhit)
misses (mmis)
prefetch
data
hits (phit)
misses (pmis)
metadata
hits (? missing from smartos' arcstat ?)
misses (? missing from smartos' arcstat ?)
> Therefore, my tentative conclusion is that memory is being paged out from the
> fileserver and being given to the ARC.
The ARC never takes memory from other consumers, it only takes from the free
list.
>
> This is the output of memstat
>
> Kernel 3612239 14110 22%
> ZFS File Data 1474454 5759 9%
> Anon 11110378 43399 66%
The anon pages are the only pages that can be swapped out. For a system like
this,
you will want to make sure your swap devices are at least this big, otherwise
the swap
memory is reserved from real memory -- a 2x hit.
-- richard
> Exec and libs 2429 9 0%
> Page cache 14309 55 0%
> Free (cachelist) 11307 44 0%
> Free (freelist) 535660 2092 3%
>
> Any suggestions on how I can confirm that this is indeed what's going on and
> how I can fix it? One solution that would work for me would be to dedicate
> say 4-6GB of RAM to the fileserver zone but I can't seem to find an option in
> vmadm to do that.
>
> Thanks,
> Rajesh
>
> smartos-discuss | Archives | Modify Your Subscription
--
[email protected]
+1-760-896-4422
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription:
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
