Hi,
we're using vibe-d (on Linux) for a long running REST API server
and have problem with constantly growing memory until system
kills it with OOM killer.
First guess was some memory leak so we've added periodic call to:
```D
GC.collect();
GC.minimize();
malloc_trim(0);
```
And when called very often (ie 500ms) service memory stays stable
and doesn't grow, so it doesn't look as a memory leak (or at
least not that fast to explain that service can go from 90MB to
900MB in 2 days).
But this is very bad for performance so we've prolonged the
interval for example to 30 seconds and now memory still goes up
(not that dramatically but still).
Stats of the GC when it growed up between 30s are these (logged
after `GC.collect()` and `GC.minimize()`:
```
GC.stats: usedSize=9994864, freeSize=92765584, total=102760448
GC.stats: usedSize=11571456, freeSize=251621120, total=263192576
```
Before the grow it has 88MB free space from 98MB total allocated.
After 30s it has 239MB free from 251MB allocated.
So it wastes a lot of free space which it can't return back to OS
for some reason.
Can these numbers be caused by memory fragmentation? There is
probably a lot of small allocations (postgresql query, result
processing and REST API json serialization).
Only explanation that makes some sense is that in some operation
there is required memory allocations that can't be fulfilled with
current memory pool (ie due to the memory fragmentation in it)
and then it allocates some data in new memory segment that can't
be returned afterwards as it still holds the 'live' data. But
that should be freed too at some point and GC should minimize (if
another request doesn'cause allocation in the 'wrong' page again).
But still, the amount of used vs free memory seems wrong, its a
whooping 95% of free space that can't be minimized :-o. I have a
problem imagining some fragmentation in it.
Are there any tools that can help diagnosing this more?
Also note that `malloc_trim` is not called from the GC itself and
as internally used malloc handles it's own memory pool it has
it's own quirks with returning unused memory back to the OS (it
does it only on `free()` in some cases).
See for example:
http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html
Behavior of malloc can be controlled with `mallopt` with:
```
M_TRIM_THRESHOLD
When the amount of contiguous free memory at the top of
the heap grows sufficiently large, free(3) employs sbrk(2) to
release this memory back to the system. (This can be useful in
programs that continue to execute
for a long period after freeing a significant amount of
memory.) The M_TRIM_THRESHOLD parameter specifies the minimum
size (in bytes) that this block of memory must reach before
sbrk(2) is used to trim the heap.
The default value for this parameter is 128*1024.
```
But the default is 128kB free heap memory block for trim to
activate but when `malloc_trim` is called manually, a much larger
block of memory is often returned. Thats puzzling too :)
