Re: GC memory fragmentation

2021-11-03 Thread Imperatorn via Digitalmars-d-learn
On Wednesday, 3 November 2021 at 06:26:49 UTC, Heromyth wrote: On Wednesday, 14 April 2021 at 12:47:22 UTC, Heromyth wrote: On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing

Re: GC memory fragmentation

2021-11-02 Thread Heromyth via Digitalmars-d-learn
On Wednesday, 14 April 2021 at 12:47:22 UTC, Heromyth wrote: On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. The Hunt Fram

Re: GC memory fragmentation

2021-04-17 Thread kdevel via Digitalmars-d-learn
On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: we're using vibe-d (on Linux) for a long running REST API server [...] [...] lot of small allocations (postgresql query, result processing and REST API json serialization). I am wondering about your overall system design. Are there

Re: GC memory fragmentation

2021-04-16 Thread sarn via Digitalmars-d-learn
On Tuesday, 13 April 2021 at 12:30:13 UTC, tchaloupka wrote: Some kind of GC memory dump and analyzer tool as mentioned `Diamond` would be of tremendous help to diagnose this.. I've used bpftrace to do some of that stuff: https://theartofmachinery.com/2019/04/26/bpftrace_d_gc.html

Re: GC memory fragmentation

2021-04-14 Thread Imperatorn via Digitalmars-d-learn
On Wednesday, 14 April 2021 at 12:47:22 UTC, Heromyth wrote: On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. The Hunt Fram

Re: GC memory fragmentation

2021-04-14 Thread Heromyth via Digitalmars-d-learn
On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. The Hunt Framework is also suffering from this. We are trying to make a si

Re: GC memory fragmentation

2021-04-14 Thread tchaloupka via Digitalmars-d-learn
On Tuesday, 13 April 2021 at 12:30:13 UTC, tchaloupka wrote: I'm not so sure if pages of small objects (or large) that are not completely empty can be reused as a new bucket or only free pages can be reused. Does anyone has some insight of this? Some kind of GC memory dump and analyzer tool a

Re: GC memory fragmentation

2021-04-13 Thread Tobias Pankrath via Digitalmars-d-learn
On Tuesday, 13 April 2021 at 12:30:13 UTC, tchaloupka wrote: Some kind of GC memory dump and analyzer tool as mentioned `Diamond` would be of tremendous help to diagnose this.. You could try to get the stack traces of the allocating calls via eBPF. Maybe that leads to some new insights.

Re: GC memory fragmentation

2021-04-13 Thread tchaloupka via Digitalmars-d-learn
On Monday, 12 April 2021 at 07:03:02 UTC, Sebastiaan Koppe wrote: We have similar problems, we see memory usage alternate between plateauing and then slowly growing. Until it hits the configured maximum memory for that job and the orchestrator kills it (we run multiple instances and have good

Re: GC memory fragmentation

2021-04-12 Thread Per Nordlöw via Digitalmars-d-learn
On Monday, 12 April 2021 at 20:50:49 UTC, Per Nordlöw wrote: more aggressive collections when nearing it. What is a more aggressive collection compared to a normal collection? Unfortunately we still have no move support in D's GC because of its impreciseness.

Re: GC memory fragmentation

2021-04-12 Thread Per Nordlöw via Digitalmars-d-learn
On Monday, 12 April 2021 at 20:50:49 UTC, Per Nordlöw wrote: I'm surprised there is no such functionality available. It doesn't sound to me like it's that difficult to implement. Afaict, we're looking for a way to call `GC.collect()` when `GC.Status.usedSize` [1] reaches a certain threshold.

Re: GC memory fragmentation

2021-04-12 Thread Per Nordlöw via Digitalmars-d-learn
On Monday, 12 April 2021 at 07:03:02 UTC, Sebastiaan Koppe wrote: On side-note, it would also be good if the GC can be aware of the max memory it is allotted so that it knows it needs to do more aggressive collections when nearing it. I'm surprised there is no such functionality available. It

Re: GC memory fragmentation

2021-04-12 Thread Imperatorn via Digitalmars-d-learn
On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. [...] Looks like the GC needs some love

Re: GC memory fragmentation

2021-04-12 Thread Sebastiaan Koppe via Digitalmars-d-learn
On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. [...] But this is very bad for performance so we've prolonged the interval

Re: GC memory fragmentation

2021-04-11 Thread frame via Digitalmars-d-learn
On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. Do you have a manual GC.free() in your code, maybe with a larger array?

Re: GC memory fragmentation

2021-04-11 Thread ryuukk_ via Digitalmars-d-learn
I should have added https://dub.pm/package-format-json#build-types profileGC as build option in your dub file That will generate a profile_gc file once the program exit, a table that lists all the allocations

Re: GC memory fragmentation

2021-04-11 Thread ryuukk_ via Digitalmars-d-learn
On Sunday, 11 April 2021 at 13:50:12 UTC, tchaloupka wrote: On Sunday, 11 April 2021 at 12:20:39 UTC, Nathan S. wrote: One thing that comes to mind: is your application compiled as 32-bit? The garbage collector is much more likely to leak memory with a 32-bit address space since it much more

Re: GC memory fragmentation

2021-04-11 Thread tchaloupka via Digitalmars-d-learn
On Sunday, 11 April 2021 at 12:20:39 UTC, Nathan S. wrote: One thing that comes to mind: is your application compiled as 32-bit? The garbage collector is much more likely to leak memory with a 32-bit address space since it much more likely for a random int to appear to be a pointer to the int

Re: GC memory fragmentation

2021-04-11 Thread Nathan S. via Digitalmars-d-learn
On Sunday, 11 April 2021 at 09:10:22 UTC, tchaloupka wrote: Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. One thing that comes to mind: is your application compiled as 32-bit? The ga

GC memory fragmentation

2021-04-11 Thread tchaloupka via Digitalmars-d-learn
Hi, we're using vibe-d (on Linux) for a long running REST API server and have problem with constantly growing memory until system kills it with OOM killer. First guess was some memory leak so we've added periodic call to: ```D GC.collect(); GC.minimize(); malloc_trim(0); ``` And when called