Re: High memory usage in vibe.d application
On Sunday, 1 July 2018 at 20:15:02 UTC, crimaniak wrote:
On Sunday, 1 July 2018 at 13:44:23 UTC, Anton Fediushin wrote:
I reduced the test case to _one_ line:
```
1.seconds.setTimer(() =>
"http://google.com".requestHTTP((scope req) {}, (scope res)
{res.disconnect;}), true);
```
What happens is `res.disconnect` doesn't free all of the
internal buffers, causing leakage. One way to avoid that is to
call `res.dropBody`, but it isn't always wanted (just like in
my example).
The problem is known and mentioned in the documentation:
http://vibed.org/api/vibe.http.client/requestHTTP
Note that it is highly recommended to use one of the overloads
that take a responder callback, as they can avoid some memory
allocations and are safe against accidentally leaving stale
response objects (objects whose response body wasn't fully
read). For the returning overloads of the function it is
recommended to put a scope(exit) right after the call in which
HTTPClientResponse.dropBody is called to avoid this.
As I understand the situation, request object will reside in
memory until you fully read or do something with response body.
It says so "for the returning overloads". Callback-based ones
should be "safe against accidentally leaving stale response
objects". Actually, in this example I don't 'accidentally' leave
objects, I do that on purpose and call `res.disconnect` to
forcefully close everything. Yet it still doesn't free memory.
There's nothing much to do with the response body - it can be
either read and destroyed or just destroyed, and `res.disconect`
should do this.
Re: High memory usage in vibe.d application
On Sunday, 1 July 2018 at 13:44:23 UTC, Anton Fediushin wrote:
I reduced the test case to _one_ line:
```
1.seconds.setTimer(() => "http://google.com".requestHTTP((scope
req) {}, (scope res) {res.disconnect;}), true);
```
What happens is `res.disconnect` doesn't free all of the
internal buffers, causing leakage. One way to avoid that is to
call `res.dropBody`, but it isn't always wanted (just like in
my example).
The problem is known and mentioned in the documentation:
http://vibed.org/api/vibe.http.client/requestHTTP
Note that it is highly recommended to use one of the overloads
that take a responder callback, as they can avoid some memory
allocations and are safe against accidentally leaving stale
response objects (objects whose response body wasn't fully
read). For the returning overloads of the function it is
recommended to put a scope(exit) right after the call in which
HTTPClientResponse.dropBody is called to avoid this.
As I understand the situation, request object will reside in
memory until you fully read or do something with response body.
Re: High memory usage in vibe.d application
On Sunday, 1 July 2018 at 12:32:25 UTC, Jacob Shtokolov wrote:
On Sunday, 1 July 2018 at 05:20:17 UTC, Anton Fediushin wrote:
Now I tried it and indeed, it's vibe.d's fault. I'm not quite
sure what causes it and if this problem is known, I'll look
into that later and open an issue if it doesn't exist already.
Yes, please do this when you have time. That would be really
helpful for further vibe.d improvement.
I remember a pretty old (and closed) bug of HTTP client here:
https://github.com/vibe-d/vibe.d/issues/1321
So it might be somehow related to this one. Probably something
wrong with HTTP client or TLS/SSL related logic. You example
code is very good and I was able to reproduce the same issue
with the latest stable compiler, so I hope the guys will find
the problem.
Thanks!
I reduced the test case to _one_ line:
```
1.seconds.setTimer(() => "http://google.com".requestHTTP((scope
req) {}, (scope res) {res.disconnect;}), true);
```
What happens is `res.disconnect` doesn't free all of the internal
buffers, causing leakage. One way to avoid that is to call
`res.dropBody`, but it isn't always wanted (just like in my
example).
I submitted an issue: https://github.com/vibe-d/vibe.d/issues/2179
Re: High memory usage in vibe.d application
On Sunday, 1 July 2018 at 05:20:17 UTC, Anton Fediushin wrote: Now I tried it and indeed, it's vibe.d's fault. I'm not quite sure what causes it and if this problem is known, I'll look into that later and open an issue if it doesn't exist already. Yes, please do this when you have time. That would be really helpful for further vibe.d improvement. I remember a pretty old (and closed) bug of HTTP client here: https://github.com/vibe-d/vibe.d/issues/1321 So it might be somehow related to this one. Probably something wrong with HTTP client or TLS/SSL related logic. You example code is very good and I was able to reproduce the same issue with the latest stable compiler, so I hope the guys will find the problem. Thanks!
Re: High memory usage in vibe.d application
On Saturday, 30 June 2018 at 22:06:50 UTC, Jacob Shtokolov wrote: On Friday, 29 June 2018 at 17:40:07 UTC, Anton Fediushin wrote: So, long story short: - Usage of Mallocator instead of theAllocator made it a little bit better - VibeManualMemoryManagement had no (or little) effect - Manually calling GC.collect had no (or little) effect You could try to call GC.minimize in pair with GC.collect: ``` GC.collect(); GC.minimize(); ``` to return all freed memory back to the OS. With vibe.d this had no effect too. Not sure that the leakage of this type is possible because if you're running your program on 64bit Linux the probability of it is very low. AFAIK the GC is launched every (almost) time you allocate the memory, and if it finds "dead" pointers, it definitely must clean them out. Vibe.d may also leak. Have you tried to run the same code without Vibe.d, say, using https://github.com/ikod/dlang-requests as an HTTP client? Now I tried it and indeed, it's vibe.d's fault. I'm not quite sure what causes it and if this problem is known, I'll look into that later and open an issue if it doesn't exist already.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 17:40:07 UTC, Anton Fediushin wrote: So, long story short: - Usage of Mallocator instead of theAllocator made it a little bit better - VibeManualMemoryManagement had no (or little) effect - Manually calling GC.collect had no (or little) effect You could try to call GC.minimize in pair with GC.collect: ``` GC.collect(); GC.minimize(); ``` to return all freed memory back to the OS. Not sure that the leakage of this type is possible because if you're running your program on 64bit Linux the probability of it is very low. AFAIK the GC is launched every (almost) time you allocate the memory, and if it finds "dead" pointers, it definitely must clean them out. Vibe.d may also leak. Have you tried to run the same code without Vibe.d, say, using https://github.com/ikod/dlang-requests as an HTTP client? Also, have you tried to change vibe.d's event loop engine, like libevent or libasync?
Re: High memory usage in vibe.d application
On 30/06/2018 7:42 PM, Anton Fediushin wrote: On Saturday, 30 June 2018 at 05:00:35 UTC, rikki cattermole wrote: On 30/06/2018 4:49 AM, Bauss wrote: I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory. Let's be honest, I don't think it was meant to live in a container with 64mb of ram. I just don't think it is kicking in to collect. It doesn't, I'm experimenting with different GC configurations [1]. By default [2] `maxPoolSize` is set to 64MB, so maybe program gets killed by docker right before GC decides to collect. [1] https://dlang.org/spec/garbage.html#gc_config [2] https://github.com/dlang/druntime/blob/master/src/gc/config.d#L23 The OS ext. takes memory too. 34mb might be safer.
Re: High memory usage in vibe.d application
On Saturday, 30 June 2018 at 05:00:35 UTC, rikki cattermole wrote: On 30/06/2018 4:49 AM, Bauss wrote: I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory. Let's be honest, I don't think it was meant to live in a container with 64mb of ram. I just don't think it is kicking in to collect. It doesn't, I'm experimenting with different GC configurations [1]. By default [2] `maxPoolSize` is set to 64MB, so maybe program gets killed by docker right before GC decides to collect. [1] https://dlang.org/spec/garbage.html#gc_config [2] https://github.com/dlang/druntime/blob/master/src/gc/config.d#L23
Re: High memory usage in vibe.d application
On 30/06/2018 4:49 AM, Bauss wrote: I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory. Let's be honest, I don't think it was meant to live in a container with 64mb of ram. I just don't think it is kicking in to collect.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 16:49:41 UTC, Bauss wrote: On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote: On 29/06/2018 11:09 PM, Anton Fediushin wrote: It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory? Probably doesn't know that it should deallocate so eagerly. A GC.collect(); call may help. That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage. Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back. I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory. I am not quite sure what should I blame now, because even if I use malloc for memory allocation, memory goes... somewhere? So, long story short: - Usage of Mallocator instead of theAllocator made it a little bit better - VibeManualMemoryManagement had no (or little) effect - Manually calling GC.collect had no (or little) effect It makes me think that error is somewhere else. I made a code snippet of my testing program: https://gitlab.com/snippets/1729304 There are some changes to it: - It uses different stream with metaint of 32KB - It calls nowPlaying() every second Now I will take a break from this, dealing with this kind of nonsense annoys me.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 16:19:39 UTC, 12345swordy wrote: On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote: Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back. Which language that you had write apps in that utilize GC? Java? C#? You shouldn't treat D GC the same as other languages GC. Alexander Talking about D here. GC can be the best option for some languages and environments, but it doesn't fit D that well. Writing programs in D I always know where stack-allocated structs get deleted and such, but I have no idea on what's going on with the GC. Does it collect anything at all? Why doesn't it collect this? How do I force it to collect this?
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote: On 29/06/2018 11:09 PM, Anton Fediushin wrote: It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory? Probably doesn't know that it should deallocate so eagerly. A GC.collect(); call may help. That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage. Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back. I wouldn't really blame the GC. There is a higher chance you're just not using it how it's meant to be, especially since it looks like you're mixing manual memory management with GC memory.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote: On 29/06/2018 11:09 PM, Anton Fediushin wrote: It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory? Probably doesn't know that it should deallocate so eagerly. A GC.collect(); call may help. That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage. Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back. Which language that you had write apps in that utilize GC? Java? C#? You shouldn't treat D GC the same as other languages GC. Alexander
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote: On 29/06/2018 11:09 PM, Anton Fediushin wrote: It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory? Probably doesn't know that it should deallocate so eagerly. A GC.collect(); call may help. That's a good idea. GC really needs to be kicked in once in a while because it did _nothing_ in 8 hours, even though my application is just a couple of timers - it isn't a hard task for CPU or memory and there's plenty of time to collect some garbage. Now I finally understand why GC is not a great thing. I was writing apps utilizing GC for a long time and never had problems with it, but when it came down to this simple program it stabbed me in the back.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 14:10:26 UTC, Daniel Kozak wrote: Have you try use VibeManualMemoryManagement https://github.com/TechEmpower/FrameworkBenchmarks/blob/3b24d0a21463edc536b30e2cea647fd425915401/frameworks/D/vibed/dub.json#L22 I'll try, not quite sure it'll help much.
Re: High memory usage in vibe.d application
Have you try use VibeManualMemoryManagement https://github.com/TechEmpower/FrameworkBenchmarks/blob/3b24d0a21463edc536b30e2cea647fd425915401/frameworks/D/vibed/dub.json#L22 On Fri, Jun 29, 2018 at 3:20 PM Anton Fediushin via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > On Friday, 29 June 2018 at 11:42:18 UTC, bauss wrote: > > On Friday, 29 June 2018 at 11:24:14 UTC, Anton Fediushin wrote: > >> On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote: > >>> On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote: > On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin > wrote: > > Almost forgot, there are two timers which call this > > function for two different streams. > > > > Value of `metaint` is 16000, which means that only 16KB of > > memory are allocated for the `buffer`, then it reads > > another byte which contains length of the metadata / 16 and > > then it reads the metadata which is 100-200 bytes long. > > > > This gives us... 16KiB per one nowPlaying() call. Why > > doesn't it free the memory? > > Maybe use the > https://dlang.org/phobos/std_experimental_allocator_mallocator.html > instead of theAllocator as it defaults to GC. > >>> > >>> Thanks, I'll try that. > >>> ... > >>> I will deploy that and see if it changes anything. > >> > >> It did! Memory usage went down to 7MiB yet it still grows > >> slightly. I'll monitor if it changes in a couple of hours but > >> it is much better. > >> > >> Thank you a lot, Radu. It turns out that theAllocator is so > >> tricky. > > > > Again you could do @nogc and see what memory is possibly > > allocated by the GC and perhaps that way you can see what > > memory the GC is holding on to. > > @nogc tells nothing new, just an error on every single line > because neither `res.bodyReader.read` nor Mallocator's functions > are marked as @nogc. > > Compiling with dmd's `-vgc` flag shows nothing but the last line. > > > > > non-GC memory should be freed right away and those there > > shouldn't be a leak from that. > > Using Mallocator instead of theAllocator improved the situation, > but it still leaks for some reason. After 2 hours it went from > 7MiB to 18MiB. > > I will compile it with profile-gc again and look for the possible > cause of that, maybe I'll try valgrind too. > > >
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 11:42:18 UTC, bauss wrote: On Friday, 29 June 2018 at 11:24:14 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote: On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote: Almost forgot, there are two timers which call this function for two different streams. Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long. This gives us... 16KiB per one nowPlaying() call. Why doesn't it free the memory? Maybe use the https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of theAllocator as it defaults to GC. Thanks, I'll try that. ... I will deploy that and see if it changes anything. It did! Memory usage went down to 7MiB yet it still grows slightly. I'll monitor if it changes in a couple of hours but it is much better. Thank you a lot, Radu. It turns out that theAllocator is so tricky. Again you could do @nogc and see what memory is possibly allocated by the GC and perhaps that way you can see what memory the GC is holding on to. @nogc tells nothing new, just an error on every single line because neither `res.bodyReader.read` nor Mallocator's functions are marked as @nogc. Compiling with dmd's `-vgc` flag shows nothing but the last line. non-GC memory should be freed right away and those there shouldn't be a leak from that. Using Mallocator instead of theAllocator improved the situation, but it still leaks for some reason. After 2 hours it went from 7MiB to 18MiB. I will compile it with profile-gc again and look for the possible cause of that, maybe I'll try valgrind too.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 11:24:14 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote: On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote: Almost forgot, there are two timers which call this function for two different streams. Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long. This gives us... 16KiB per one nowPlaying() call. Why doesn't it free the memory? Maybe use the https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of theAllocator as it defaults to GC. Thanks, I'll try that. ... I will deploy that and see if it changes anything. It did! Memory usage went down to 7MiB yet it still grows slightly. I'll monitor if it changes in a couple of hours but it is much better. Thank you a lot, Radu. It turns out that theAllocator is so tricky. Again you could do @nogc and see what memory is possibly allocated by the GC and perhaps that way you can see what memory the GC is holding on to. non-GC memory should be freed right away and those there shouldn't be a leak from that.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote: On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote: On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote: Almost forgot, there are two timers which call this function for two different streams. Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long. This gives us... 16KiB per one nowPlaying() call. Why doesn't it free the memory? Maybe use the https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of theAllocator as it defaults to GC. Thanks, I'll try that. ... I will deploy that and see if it changes anything. It did! Memory usage went down to 7MiB yet it still grows slightly. I'll monitor if it changes in a couple of hours but it is much better. Thank you a lot, Radu. It turns out that theAllocator is so tricky.
Re: High memory usage in vibe.d application
On 29/06/2018 11:09 PM, Anton Fediushin wrote: It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory? Probably doesn't know that it should deallocate so eagerly. A GC.collect(); call may help.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 10:31:14 UTC, bauss wrote: On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote: On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote: Almost forgot, there are two timers which call this function for two different streams. Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long. This gives us... 16KiB per one nowPlaying() call. Why doesn't it free the memory? Maybe use the https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of theAllocator as it defaults to GC. Also, why you .idup the array? .array already creates a new one on the heap. This. Which kind of makes the usage of theAllocator useless. Indeed, because it uses GC by default my `theAllocator.dispose` did nothing, which basically made these two samples of code equal. I was going to suggest using @nogc too, because it would most likely be GC allocated memory that is taking up space. It is GC's fault for sure, I built my program with profile-gc and it allocated a lot there. Question is, why doesn't it free this memory? I run multiple vibe.d applications and I have no issues with memory (Even with GC.) Me neither, my other vibe.d project uses 7.5MB and that's it.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:
On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote:
Almost forgot, there are two timers which call this function
for two different streams.
Value of `metaint` is 16000, which means that only 16KB of
memory are allocated for the `buffer`, then it reads another
byte which contains length of the metadata / 16 and then it
reads the metadata which is 100-200 bytes long.
This gives us... 16KiB per one nowPlaying() call. Why doesn't
it free the memory?
Maybe use the
https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of theAllocator as it defaults to GC.
Thanks, I'll try that.
Also, why you .idup the array? .array already creates a new one
on the heap.
It does, but it creates char[] and I need a string. I changed
code a little bit to remove unnecessary `map` and `idup` too.
Code now:
```
@safe string nowPlaying(string url) {
import vibe.core.stream;
import std.experimental.allocator;
import std.experimental.allocator.mallocator;
import std.string;
string r;
url.requestHTTP(
(scope req) {
req.headers.addField("Icy-MetaData", "1");
},
(scope res) {
RCIAllocator a = allocatorObject(Mallocator.instance);
auto metaint = res.headers.get("icy-metaint").to!int;
auto buffer = a.makeArray!ubyte(metaint);
scope(exit) a.dispose(buffer);
res.bodyReader.read(buffer, IOMode.all);
auto lengthBuffer = a.makeArray!ubyte(1);
scope(exit) a.dispose(lengthBuffer);
res.bodyReader.read(lengthBuffer, IOMode.all);
auto dataBuffer = a.makeArray!ubyte(lengthBuffer[0] *
16);
scope(exit) a.dispose(dataBuffer);
res.bodyReader.read(dataBuffer, IOMode.all);
r =
dataBuffer.split('\'').drop(1).front.array.assumeUTF;
res.disconnect;
}
);
return r;
}
```
I will deploy that and see if it changes anything.
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote: On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote: Almost forgot, there are two timers which call this function for two different streams. Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long. This gives us... 16KiB per one nowPlaying() call. Why doesn't it free the memory? Maybe use the https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of theAllocator as it defaults to GC. Also, why you .idup the array? .array already creates a new one on the heap. This. Which kind of makes the usage of theAllocator useless. I was going to suggest using @nogc too, because it would most likely be GC allocated memory that is taking up space. I run multiple vibe.d applications and I have no issues with memory (Even with GC.)
Re: High memory usage in vibe.d application
On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote: Almost forgot, there are two timers which call this function for two different streams. Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long. This gives us... 16KiB per one nowPlaying() call. Why doesn't it free the memory? Maybe use the https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of theAllocator as it defaults to GC. Also, why you .idup the array? .array already creates a new one on the heap.
Re: High memory usage in vibe.d application
Almost forgot, there are two timers which call this function for two different streams. Value of `metaint` is 16000, which means that only 16KB of memory are allocated for the `buffer`, then it reads another byte which contains length of the metadata / 16 and then it reads the metadata which is 100-200 bytes long. This gives us... 16KiB per one nowPlaying() call. Why doesn't it free the memory?
High memory usage in vibe.d application
Hello, I'm looking for an advice on what I am doing wrong.
I have a vibe.d-based program, which connects to an audio stream
and gets name of the song currently playing. For that, I wrote
the following code:
```
@safe string nowPlaying(string url) {
import vibe.core.stream;
string r;
url.requestHTTP(
(scope req) {
req.headers.addField("Icy-MetaData", "1");
},
(scope res) {
auto metaint = res.headers.get("icy-metaint").to!int;
auto buffer = new ubyte[metaint];
res.bodyReader.read(buffer, IOMode.all);
auto lengthBuff = new ubyte[1];
res.bodyReader.read(lengthBuff, IOMode.all);
auto dataBuffer = new ubyte[lengthBuff[0] * 16];
res.bodyReader.read(dataBuffer, IOMode.all);
r = dataBuffer.map!(a =>
a.to!char).split('\'').drop(1).front.array.idup;
}
);
return r;
}
```
And I call it with a timer every 10 seconds:
```
string now_playing;
10.seconds.setTimer(() {
now_playing = nowPlaying(stream);
}, true);
```
This code worked fine for 8 or so hours and then got killed by
docker because of a limit of 64MB of RAM. I executed the same
code on my machine and saw resident set size growing in
real-time. Blaming GC (as people usually do) I changed the code
to use std.experimental.allocator instead:
```
@safe string nowPlaying(string url) {
import vibe.core.stream;
import std.experimental.allocator;
string r;
url.requestHTTP(
(scope req) {
req.headers.addField("Icy-MetaData", "1");
},
(scope res) {
auto metaint = res.headers.get("icy-metaint").to!int;
auto buffer = theAllocator.makeArray!ubyte(metaint);
scope(exit) theAllocator.dispose(buffer);
res.bodyReader.read(buffer, IOMode.all);
auto lengthBuffer = theAllocator.makeArray!ubyte(1);
scope(exit) theAllocator.dispose(lengthBuffer);
res.bodyReader.read(lengthBuffer, IOMode.all);
auto dataBuffer = theAllocator.makeArray!ubyte(lengthBuffer[0]
* 16);
scope(exit) theAllocator.dispose(dataBuffer);
res.bodyReader.read(dataBuffer, IOMode.all);
r = dataBuffer.map!(a =>
a.to!char).split('\'').drop(1).front.array.idup;
}
);
return r;
}
```
And somehow, it got *worse*. Now my program gets killed every 3
hours. How is that possible? Am I missing something?
Some screenshots of CPU/Memory usage:
1. These are metrics of a whole cluster, program is started at
around 8:00 and gets killed after 16:00
https://imgur.com/a/IhHvOt4
2. These are metrics of an updated program which uses
std.experimental.allocator. https://imgur.com/a/XBchJ7C
