I have some time this weekend to experiment with boehm and markandsweep.
> It wouldn't be surprising if total heap space (live objects + free space)
> went up as a result of fragmentation, but free space explosion would mean
> that you've encountered a pathological case.
"Pathological"? If dealloc() fragments a large block, then repeated allocation
of large blocks would definitely cause memory use to explode. One interesting
note: When I briefly tried boehm, it complained one or twice early in the
program about my large allocations. Perhaps the Nim mem-managers are simply not
intended for repeated alloc/dealloc of large blocks. In that case, I could try
making large allocations "threadvar", assuming I can count on data-persistence
for a thread in the threadpool. I will experiment with that this weekend.
The performance hit happens even single-threaded, not using `spawn`. I guess
it's possible that hit comes mainly from interactions with the memory manager.
Even on Linux, with the simple example I pasted above, you can verify that the
already speedy program becomes even speedier after total allocated memory
stabilizes, which indicates to me that allocations from existing freelists are
quicker than acquiring memory from the system.
I still would like to see freelist sizes. That would immediately answer
questions on memory fragmentation. GNU malloc provides far better memory
debugging tools than we have available:
[https://www.gnu.org/software/libc/manual/html_node/Using-the-Memory-Debugger.html#Using-the-Memory-Debugger](https://www.gnu.org/software/libc/manual/html_node/Using-the-Memory-Debugger.html#Using-the-Memory-Debugger)
Do you think that a pull-request for showing freelist sizes would be accepted?