Re: Cleaning/Releasing large arrays from memory
On Monday, 24 July 2017 at 18:01:19 UTC, ketmar wrote: Clinton wrote: On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote: Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free BTW, how would I do this with associative arrays? you can't. you can `.clear` 'em, though, but this won't immediately free the internal storage. Meh.. Okay, well using these two should make things a lot better. Thanks for the help and clearing up my confusion.
Re: Cleaning/Releasing large arrays from memory
Clinton wrote: On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote: Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free BTW, how would I do this with associative arrays? you can't. you can `.clear` 'em, though, but this won't immediately free the internal storage.
Re: Cleaning/Releasing large arrays from memory
On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote: Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free BTW, how would I do this with associative arrays?
Re: Cleaning/Releasing large arrays from memory
On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote: Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free Thanks! I didn't know I could do that with the GC enabled.
Re: Cleaning/Releasing large arrays from memory
rikki cattermole wrote: Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); or just `delete arr;`. it is marked as "deprecated" in changelog, but who cares? it works.
Re: Cleaning/Releasing large arrays from memory
Stuff the GC. You don't need it to care about collecting (or destroying for that matter). Tell it to free[0] the array directly. ```D T[] array; GC.free(array.ptr); ``` Normally I would suggest to create your own buffer, but because of the DB library probably doesn't support that, no point trying to force it. [0] http://dlang.org/phobos/core_memory.html#.GC.free
Cleaning/Releasing large arrays from memory
Hi guys, I have a question on how to free large arrays in D after
they're no longer needed.
Let's say I have this:
SomeKey[] getKeys() {
SomeKey[] n;
foreach(batching stuff...) {
SomeKey newstuff = db.select!(SomeKey[])(...); // gets around
6M of these from db
redis.send("newItems", newStuff.length);
n ~= newStuff;
destroy(newStuff);
}
return n;
}
void main() {
SomeKey[] myarray = getKeys();
ulong[string] mappedValues;
foreach(i, item; myArray) {
mappedValues[item.name] = i;
}
// No longer need myarray, and we need the memory back
destory(myarray);
GC.collect;
// Another memory intensive operation below
}
This seems to work for me to a point, but I notice my program
still holds a lot in memory even after certain memory intensive
arrays should have been collected.
Is this just a hit and miss with the GC or am I doing something
wrong? Uses around 40% less memory forcing the GC and cleaning
arrays but seems like a lot is left over, especially for threads
that have already finished(I've expect a thread to free all it's
memory). I'm not sharing anything across threads besides a config
struct.
This is a simplified version of my app. At the moment when at
around 9GB at it's peak instead of 19GB like before.
