Re: kmem pool for half pagesize is very wasteful
Chuck Silvers writes: > in the longer term, I think it would be good to use even larger pool pages > for large pool objects on systems that have relatively large amount of memory. > even with your patch, a 1k pool object on a system with a 4k VM page size > still has 33% overhead for the redzone, which is a lot for something that > is enabled by DIAGNOSTIC and is thus supposed to be "inexpensive". So maybe the real bug is that this check should not be part of DIAGNOSTIC. I remember from 2.8BSD that DIAGNOSTIC was basically just supposed to add cheap asserts and panic earlier but not really be slower in any way anybody would care about. It seems easy enough to make this separate and not get turned on for DIAGNOSTIC, but some other define. It might even be that for current the checked-in GENERIC enables this. But someone turning on DIAGNOSTIC on 9 shouldn't get things that hurt memory usage really at all, or more than say a 2% degradation in speed. > there's a tradeoff here in that using a pool page size that matches the > VM page size allows us to use the direct map, whereas with a larger > pool page size we can't use the direct map (at least effectively can't today), > but for pools that already use a pool page size that is larger than > the VM page size (eg. buf16k using a 64k pool page size) we already > aren't using the direct map, so there's no real penalty for increasing > the pool page size even further, as long as the larger pool page size > is still a tiny percentage of RAM and KVA. we can choose the pool page size > such that the overhead of the redzone is bounded by whatever percentage > we would like. this way we can use a redzone for most pools while > still keeping the overhead down to a reasonable level. That sounds like great progress and I don't mean to say anything negative about that. signature.asc Description: PGP signature
Re: kmem pool for half pagesize is very wasteful
On Tue, Feb 23, 2021 at 06:25:50PM +1100, matthew green wrote: > hi folks. > > > while we were debugging some memory starvation issues i noticed that > the "kmem-02048" pool only has 1 item per page on a system with 4KiB > pages, same similarly "kmem-04096" on 8KiB page systems. i assume > this also occurs on 16KiB page systems for the "kmem-08192" pool. > > this happens because the pool redzone increases the size from 2048 > bytes to 2048+CACHE_LINE_SIZE bytes. > > this feels extremely wasteful to me. for the common 4K page size > case, it's 2048+64 bytes of header of useful data, plus the 64 bytes > lost for the redzone alignment, and thus 1920 bytes of lost space. > > the lost space is similarly just under 1/2 for large page size > systems. > > for the smaller kmem pools, the lost space seems not great, but > significantly less than the previous (1024*3 + 2*64*3 = 3456 bytes > of useful info). 46.8% lost vs 15.6% lost, while not great, seems > like a reasonable compromise to almost half the memory required > for the kmem-02048 pool. > > this patch avoids this problem: > >https://www.netbsd.org/~mrg/poolwaste.diff > > comments? this patch is fine for now, please commit. in the longer term, I think it would be good to use even larger pool pages for large pool objects on systems that have relatively large amount of memory. even with your patch, a 1k pool object on a system with a 4k VM page size still has 33% overhead for the redzone, which is a lot for something that is enabled by DIAGNOSTIC and is thus supposed to be "inexpensive". there's a tradeoff here in that using a pool page size that matches the VM page size allows us to use the direct map, whereas with a larger pool page size we can't use the direct map (at least effectively can't today), but for pools that already use a pool page size that is larger than the VM page size (eg. buf16k using a 64k pool page size) we already aren't using the direct map, so there's no real penalty for increasing the pool page size even further, as long as the larger pool page size is still a tiny percentage of RAM and KVA. we can choose the pool page size such that the overhead of the redzone is bounded by whatever percentage we would like. this way we can use a redzone for most pools while still keeping the overhead down to a reasonable level. -Chuck
Re: kmem pool for half pagesize is very wasteful
m...@eterna.com.au (matthew green) writes: >while we were debugging some memory starvation issues i noticed that >the "kmem-02048" pool only has 1 item per page on a system with 4KiB >pages, same similarly "kmem-04096" on 8KiB page systems. i assume >this also occurs on 16KiB page systems for the "kmem-08192" pool. It also occurs for the buf32k pool. One item (32k) per 64k pool page. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
kmem pool for half pagesize is very wasteful
hi folks. while we were debugging some memory starvation issues i noticed that the "kmem-02048" pool only has 1 item per page on a system with 4KiB pages, same similarly "kmem-04096" on 8KiB page systems. i assume this also occurs on 16KiB page systems for the "kmem-08192" pool. this happens because the pool redzone increases the size from 2048 bytes to 2048+CACHE_LINE_SIZE bytes. this feels extremely wasteful to me. for the common 4K page size case, it's 2048+64 bytes of header of useful data, plus the 64 bytes lost for the redzone alignment, and thus 1920 bytes of lost space. the lost space is similarly just under 1/2 for large page size systems. for the smaller kmem pools, the lost space seems not great, but significantly less than the previous (1024*3 + 2*64*3 = 3456 bytes of useful info). 46.8% lost vs 15.6% lost, while not great, seems like a reasonable compromise to almost half the memory required for the kmem-02048 pool. this patch avoids this problem: https://www.netbsd.org/~mrg/poolwaste.diff comments? .mrg.