Pawel Jakub Dawidek wrote: > ZFS works really stable on FreeBSD, but I'm biggest problem is how to > control ZFS memory usage. I've no idea how to leash that beast. > > FreeBSD has a backpresure mechanism. I can register my function so it > will be called when there are memory problems, which I do. I using it > for ARC layer. > Even with this in place under heavy load the kernel panics, because > memory with KM_SLEEP cannot be allocated. > > Here are some statistics of memory usage when the panic occurs: > > zfs_znode_cache: 356 * 11547 = 4110732 bytes > zil_lwb_cache: 176 * 43 = 7568 bytes > arc_buf_t: 20 * 7060 = 141200 bytes > arc_buf_hdr_t: 188 * 7060 = 1327280 bytes > dnode_t: 756 * 162311 = 122707116 bytes !!
The physical blocks for storing dnodes are 16KB (as dictated by DNODE_BLOCK_SHIFT), with 32 dnodes per physical block. The interesting part is that if you read in one file (or have it cached by the DNLC), then that whole 16KB block has to stay in memory. So in the worst case, you have an app that somehow unluckily reads in every 32nd file / dnode (incurring the memory baggage of 31 dnodes not actively being used). eric > dmu_buf_impl_t: 332 * 18649 = 6191468 > other: 14432256 bytes (regular kmem_alloc()) > > There is 1GB of RAM, 320MB is for the kernel. 1/3 if kernel memory is > configured as ARC's maximum. > When it panics, debugger statistics show that there is around 2/3 of > this actually allocated, but probably due memory fragmentation. > > The most important part is dnode_t probably as it looks it doesn't obey > any limits. Maybe it is a bug in my port and I'm leaking them somehow? > On the other hand when I unload ZFS kernel module, FreeBSD's kernel > reports any memory leaks - they exist, but are much, much smaller. > > There is also quite a lot of znodes, which I'd also like to be able to > free and not sure how. In Solaris vnode's life end in VOP_INACTIVE() > routine, but znode if kept around. In FreeBSD VOP_INACTIVE() means "puts > the vnode onto free vnodes list" and when we want to use this vnode for > different file system VOP_RECLAIM() is called and VOP_RECLAIM() will be > a good place to free znode as well, if possible. > > Any ideas how to fix it? > > > > ------------------------------------------------------------------------ > > _______________________________________________ > zfs-code mailing list > zfs-code at opensolaris.org > http://opensolaris.org/mailman/listinfo/zfs-code
