Dennis Clarke wrote:
Justin Gombos wrote:
There is a ZFS FAQ somewhere indicating that ZFS *appears* to hog
memory because it uses as much as it can, but it supposedly
relinquishes memory as soon as an app calls for it.
I thought that should be pointed out, though I am not quick to accept
it myself. How does the kernel ask ZFS processes for more memory?
ZFS uses kmem_cache_create() and friends, which is a fun little beast.
It can manufacture and recycle pre-initialized bits of memory without
having to do much to it. One of the arguments used in setting up a
cache is a callback routine the kernel memory allocator can call when
it needs subsystems to clean up and return memory to the system.
In the particular case of the ZFS ARC cache, there's a hdr_cache and
a buf_cache; the hdr_cache has a callback, which wakes up the ARC
reclaim thread to evict unneeded allocations in several areas, not
just that one cache.
This has probably been asked before but is there a kernel tunable that
will restrict the high water mark for ZFS cache memory? This way I could
just grant 50% of mamory and no more. That sort of thing.
A more important question would be, what effect would this have on system
performance and am I merely thwarting progress. Sort of like early owners
of the automobile were forced to send someone walking in front of them
with a light in order to ensure that the horses were not frightened. That
sort of thing.
From zfs-discuss: RE:Limit ZFS Memory Utilization Mark Maybee wrote:
Ideally, there would be no need for any tunables; ZFS would always "do
the right thing". This is our grail. In the meantime, I can give some
recommendations, but there is no "rule of thumb" that is going to work
in all circumstances.
ncsize: As I have mentioned previously, there are overheads
associated with caching "vnode data" in ZFS. While
the physical on-disk data for a znode is only 512bytes,
the related in-core cost is significantly higher.
Roughly, you can expect that each ZFS vnode held in
the DNLC will cost about 3K of kernel memory.
So, you need to set ncsize appropriately for how much
memory you are willing to devote to it. 500,000 entries
is going to cost you 1.5GB of memory.
zfs_arc_max: This is the maximum amount of memory you want the
ARC to be able to use. Note that the ARC won't
necessarily use this much memory: if other applications
need memory, the ARC will shrink to accommodate.
Although, also note that the ARC *can't* shrink if all
of its memory is held. For example, data in the DNLC
cannot be evicted from the ARC, so this data must first
be evicted from the DNLC before the ARC can free up
space (this is why it is dangerous to turn off the ARCs
ability to evict vnodes from the DNLC).
Also keep in mind that the ARC size does not account for
many in-core data structures used by ZFS (znodes/dnodes/
dbufs/etc). Roughly, for every 1MB of cached file
pointers, you can expect another 3MB of memory used
outside of the ARC. So, in the example above, where
ncsize is 500,000, the ARC is only seeing about 400MB
of the 1.5GB consumed. As I have stated previously,
we consider this a bug in the current ARC accounting
that we will soon fix. This is only an issue in
environments where many files are being accessed. If
the number of files accessed is relatively low, then
the ARC size will be much closer to the actual memory
consumed by ZFS.
So, in general, you should not really need to "tune"
zfs_arc_max. However, in environments where you have
specific applications that consume known quantities of
memory (e.g. database), it will likely help to set the
ARC max size down, to guarantee that the necessary
kernel memory is available. There may be other times
when it will be beneficial to explicitly set the ARCs
maximum size, but at this time I can't offer any general
"rule of thumb".
I hope that helps.
-Mark
_______________________________________________
opensolaris-discuss mailing list
[email protected]
