Pawel Jakub Dawidek wrote:
> ZFS works really stable on FreeBSD, but I'm biggest problem is how to
> control ZFS memory usage. I've no idea how to leash that beast.
>
> FreeBSD has a backpresure mechanism. I can register my function so it
> will be called when there are memory problems, which I do. I using it
> for ARC layer.
> Even with this in place under heavy load the kernel panics, because
> memory with KM_SLEEP cannot be allocated.
>
> Here are some statistics of memory usage when the panic occurs:
>
> zfs_znode_cache: 356 * 11547 = 4110732 bytes
> zil_lwb_cache: 176 * 43 = 7568 bytes
> arc_buf_t: 20 * 7060 = 141200 bytes
> arc_buf_hdr_t: 188 * 7060 = 1327280 bytes
> dnode_t: 756 * 162311 = 122707116 bytes !!
> dmu_buf_impl_t: 332 * 18649 = 6191468
> other: 14432256 bytes (regular kmem_alloc())
>
> There is 1GB of RAM, 320MB is for the kernel. 1/3 if kernel memory is
> configured as ARC's maximum.
> When it panics, debugger statistics show that there is around 2/3 of
> this actually allocated, but probably due memory fragmentation.
>
> The most important part is dnode_t probably as it looks it doesn't obey
> any limits. Maybe it is a bug in my port and I'm leaking them somehow?
> On the other hand when I unload ZFS kernel module, FreeBSD's kernel
> reports any memory leaks - they exist, but are much, much smaller.
>
> There is also quite a lot of znodes, which I'd also like to be able to
> free and not sure how. In Solaris vnode's life end in VOP_INACTIVE()
> routine, but znode if kept around. In FreeBSD VOP_INACTIVE() means "puts
> the vnode onto free vnodes list" and when we want to use this vnode for
> different file system VOP_RECLAIM() is called and VOP_RECLAIM() will be
> a good place to free znode as well, if possible.
>
> Any ideas how to fix it?
>
The problem is that in ZFS the vnode holds onto more memory than just
the vnode itself. Its fine to place the vnode on a "free vnodes list"
after a VOP_INACTIVE()... but you need to make sure the you have
"released" the *extra* memory associated with the vnode:
vnode refs a znode (356 bytes + 512 bytes for the phys)
znode refs a dnode (756 bytes + 512-16k for the phys)
So a vnode could be holding up to 17k of data in memory!
I suggest you free up the znode at VOP_INACTIVE() rather than waiting
until VOP_RECLAIM().
-Mark
