If your memory blocks are roughly in the same "size classes" and fit into jemalloc's bucket sizes, then jemalloc (which AFAIK is used as emscripten's default allocator) should be fairly effective in avoiding memory fragmentation because allocations of similar size are already grouped into bigger pool allocations and then reused.
Fragmentation might then become a problem on the next level, when jemalloc is trying to allocate memory for new buckets, but can't find a big enough gap in the 32-bit address range (I'm actually not sure if WASM even uses the full 4 GB range, or is limited to 2 GB, in that case, 1.7 GB might be too close to the maximum, and no clever "fragmentation strategy" might help, because there's too little "wiggle room"). You can read up on jemalloc allocation strategy by searching here for "IMPLEMENTATION NOTES": http://jemalloc.net/jemalloc.3.html. On Wednesday, 24 February 2021 at 18:46:04 UTC+1 Dieter Weidenbrück wrote: > Hi all, > I'm working on a graphics authoring app. Naturally, there are lots of > primitives stored in variable size mem blocks. The quantity can easily > exceed a couple of hundred thousand blocks. > Loading such content into fresh memory is not the problem, however, after > editing/changing the content the heap will be cluttered with small blocks. > > It is not really feasible to load a new instance and reorganize all blocks > there, although that might be a last effort solution. > > Is there any recommendation on how to overcome this problem? I thought > about reserving a large chunk of memory (approx 1.7 GB to leave room for > other allocations) using malloc at startup time and then manage all small > blocks myself. Looks a bit like brute force though. (I'm not caring about > mobile phone usage) > > On 68k-Macs Handles were used (int**) where a pointer was stored in a > master block, and the code would access the location in the master block > only. That made it possible to move mem blocks around and compact the heap. > Could that be a solution? > > Thanks for your ideas! > Coming from long years of C development I am thrilled with emscripten and > the options it offers for new kinds of web apps! > -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/afa47085-85fd-437e-9e6e-94d20b5dfc1en%40googlegroups.com.
