I have some time this weekend to experiment with boehm and markandsweep.

> It wouldn't be surprising if total heap space (live objects + free space) 
> went up as a result of fragmentation, but free space explosion would mean 
> that you've encountered a pathological case.

"Pathological"? If dealloc() fragments a large block, then repeated allocation 
of large blocks would definitely cause memory use to explode. One interesting 
note: When I briefly tried boehm, it complained one or twice early in the 
program about my large allocations. Perhaps the Nim mem-managers are simply not 
intended for repeated alloc/dealloc of large blocks. In that case, I could try 
making large allocations "threadvar", assuming I can count on data-persistence 
for a thread in the threadpool. I will experiment with that this weekend.

The performance hit happens even single-threaded, not using `spawn`. I guess 
it's possible that hit comes mainly from interactions with the memory manager. 
Even on Linux, with the simple example I pasted above, you can verify that the 
already speedy program becomes even speedier after total allocated memory 
stabilizes, which indicates to me that allocations from existing freelists are 
quicker than acquiring memory from the system.

I still would like to see freelist sizes. That would immediately answer 
questions on memory fragmentation. GNU malloc provides far better memory 
debugging tools than we have available:
    
[https://www.gnu.org/software/libc/manual/html_node/Using-the-Memory-Debugger.html#Using-the-Memory-Debugger](https://www.gnu.org/software/libc/manual/html_node/Using-the-Memory-Debugger.html#Using-the-Memory-Debugger)

Do you think that a pull-request for showing freelist sizes would be accepted?

Reply via email to