Re: Garbage collection

2020-07-20 Thread James Gray via Digitalmars-d-learn

On Tuesday, 30 June 2020 at 06:16:26 UTC, Kagamin wrote:

On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
I have produced something which essentially reproduces my 
problem.


What is the problem? Do you have a leak or you want to know how 
GC works?


I have managed to resolve my problem (which was a memory leak). 
My code uses a large data structure similar to a link list and 
the garbage collector was not collecting it. However,
if I set all the "links" between the nodes in the data structure 
to null it is then collected.


Re: Garbage collection

2020-06-30 Thread Kagamin via Digitalmars-d-learn

On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:
I have produced something which essentially reproduces my 
problem.


What is the problem? Do you have a leak or you want to know how 
GC works?


Re: Garbage collection

2020-06-27 Thread James Gray via Digitalmars-d-learn

On Saturday, 27 June 2020 at 14:49:34 UTC, James Gray wrote:

On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:

[...]


Thank you for doing this. I hope my example doesn't obscure 
what you show here.

(I borrowed some of your code).

[...]


In case it helps, setting all the next and previous pointers in 
the link list to null allows the garbage collector to collect in 
the above code.


Re: Garbage collection

2020-06-27 Thread Stanislav Blinov via Digitalmars-d-learn

On Saturday, 27 June 2020 at 16:03:12 UTC, kinke wrote:
On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov 
wrote:



Hrm... What happens if you call collect() twice?


Nothing changes, even when collecting 5 times at the end of 
each iteration. In the filed testcase, I've extracted the stack 
ref to a dedicated function, so that there really shouldn't be 
any refs on the stack (this is unoptimized code after all...).


Here on Linux, the double collection results in this output:

  GC stats: 0M used, 0M free, 0M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total
Starting
  string size: 943M
  GC stats: 0M used, 2306M free, 2306M total


Re: Garbage collection

2020-06-27 Thread kinke via Digitalmars-d-learn

On Saturday, 27 June 2020 at 15:27:34 UTC, Stanislav Blinov wrote:

On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:

Note that I explicitly clear the `str` slice before 
GC.collect(), so that the stack shouldn't contain any refs to 
the fat string anymore.


Hrm... What happens if you call collect() twice?


Nothing changes, even when collecting 5 times at the end of each 
iteration. In the filed testcase, I've extracted the stack ref to 
a dedicated function, so that there really shouldn't be any refs 
on the stack (this is unoptimized code after all...).


Re: Garbage collection

2020-06-27 Thread Stanislav Blinov via Digitalmars-d-learn

On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:

Note that I explicitly clear the `str` slice before 
GC.collect(), so that the stack shouldn't contain any refs to 
the fat string anymore.


Hrm... What happens if you call collect() twice?


Re: Garbage collection

2020-06-27 Thread James Gray via Digitalmars-d-learn

On Saturday, 27 June 2020 at 14:12:09 UTC, kinke wrote:

On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:

have run into a memory leak


Something seems really off indeed. I've run this on Win64 with 
DMD (2.092) and LDC (1.22), without any extra cmdline options:


-
import core.memory;
import core.stdc.stdio;
import std.range;
import std.format;

auto f(R)(R r) { return format("%s", r); }

int toMB(ulong size) { return cast(int) (size / 1048576.0 + 
0.5); }


void printGCStats()
{
const stats = GC.stats;
const used = toMB(stats.usedSize);
const free = toMB(stats.freeSize);
const total = toMB(stats.usedSize + stats.freeSize);
printf("  GC stats: %dM used, %dM free, %dM total\n", used, 
free, total);

}

void main()
{
printGCStats();

while (true)
{
puts("Starting");
string str = f(iota(100_000_000));
printf("  string size: %dM\n", toMB(str.length));
str = null;
GC.collect();
printGCStats();
}
}
-

Output with DMD (no change with the precise GC via 
`--DRT-gcopt=gc:precise`):

-
  GC stats: 0M used, 1M free, 1M total
Starting
  string size: 943M
  GC stats: 1168M used, 1139M free, 2306M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
-

With LDC:
-
  GC stats: 0M used, 1M free, 1M total
Starting
  string size: 943M
  GC stats: 1168M used, 1139M free, 2306M total
Starting
  string size: 943M
  GC stats: 2335M used, 1288M free, 3623M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
-

Note that I explicitly clear the `str` slice before 
GC.collect(), so that the stack shouldn't contain any refs to 
the fat string anymore.


Thank you for doing this. I hope my example doesn't obscure what 
you show here.

(I borrowed some of your code).

I have produced something which essentially reproduces my problem.

---
import std.range;
import std.algorithm;
import std.format;
import std.stdio;
import core.thread;
import core.memory;


struct Node {
 Node* next;
 Node* prev;
 ulong val;
}

Node* insertAfter(Node* cur, ulong val) {
 Node* node = new Node;
 if (cur != null) {
  node.next = cur.next;
  node.prev = cur;
  cur.next = node;
  node.next.prev = node;
 }
 else {
  node.next = node;
  node.prev = node;
 }
 node.val = val;
 return node;
}

int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); 
}


void printGCStats()
{
 const stats = GC.stats;
 const used = toMB(stats.usedSize);
 const free = toMB(stats.freeSize);
 const total = toMB(stats.usedSize + stats.freeSize);
 writef("  GC stats: %dM used, %dM free, %dM total\n", used, 
free, total);

}


void main()
{
 while(true)
 {
  printGCStats();
  writeln("Starting");
  Node* dll;
  dll = iota(2).fold!((c,x)=>insertAfter(c,x))(dll);
  writef("Last element %s\n", dll.val);
  dll = null;
  writeln("Done");
  GC.collect();
  GC.minimize();
  Thread.sleep( dur!("msecs")( 1 ) );
 }
}
--
With DMD this produces:

  GC stats: 0M used, 0M free, 0M total
Starting
Last element 1
Done
  GC stats: 6104M used, 51M free, 6155M total
Starting
Last element 1
Done
  GC stats: 12207M used, 28M free, 12235M total

With LDC2 this produces:

  GC stats: 0M used, 0M free, 0M total
Starting
Last element 1
Done
  GC stats: 6104M used, 51M free, 6155M total
Starting
Last element 1
Done
  GC stats: 12207M used, 28M free, 12235M total







Re: Garbage collection

2020-06-27 Thread kinke via Digitalmars-d-learn

=> https://issues.dlang.org/show_bug.cgi?id=20983


Re: Garbage collection

2020-06-27 Thread kinke via Digitalmars-d-learn

On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:

have run into a memory leak


Something seems really off indeed. I've run this on Win64 with 
DMD (2.092) and LDC (1.22), without any extra cmdline options:


-
import core.memory;
import core.stdc.stdio;
import std.range;
import std.format;

auto f(R)(R r) { return format("%s", r); }

int toMB(ulong size) { return cast(int) (size / 1048576.0 + 0.5); 
}


void printGCStats()
{
const stats = GC.stats;
const used = toMB(stats.usedSize);
const free = toMB(stats.freeSize);
const total = toMB(stats.usedSize + stats.freeSize);
printf("  GC stats: %dM used, %dM free, %dM total\n", used, 
free, total);

}

void main()
{
printGCStats();

while (true)
{
puts("Starting");
string str = f(iota(100_000_000));
printf("  string size: %dM\n", toMB(str.length));
str = null;
GC.collect();
printGCStats();
}
}
-

Output with DMD (no change with the precise GC via 
`--DRT-gcopt=gc:precise`):

-
  GC stats: 0M used, 1M free, 1M total
Starting
  string size: 943M
  GC stats: 1168M used, 1139M free, 2306M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
Starting
  string size: 943M
  GC stats: 1168M used, 2456M free, 3623M total
-

With LDC:
-
  GC stats: 0M used, 1M free, 1M total
Starting
  string size: 943M
  GC stats: 1168M used, 1139M free, 2306M total
Starting
  string size: 943M
  GC stats: 2335M used, 1288M free, 3623M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
Starting
  string size: 943M
  GC stats: 2335M used, 2605M free, 4940M total
-

Note that I explicitly clear the `str` slice before GC.collect(), 
so that the stack shouldn't contain any refs to the fat string 
anymore.


Re: Garbage collection

2020-06-27 Thread James Gray via Digitalmars-d-learn

On Saturday, 27 June 2020 at 12:07:19 UTC, Stanislav Blinov wrote:

On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote:

If you are using linux, have in mind that the memory is often 
not returned to the OS even after a (libc) free.


That's a good observation. Although a GC implementation is not 
required to actually use malloc, so depending on that falls 
into "know what you're doing" territory :)


Thanks for the help, but unfortunately it isn't stopping memory 
usage growing in the original app. I will try and build a minimal 
example. In the meantime perhaps someone can suggest how I might 
figure out what is going on. Repeating the same action is giving 
memory usage growth as follows. 1.7GB first time (which now drops 
to about 1GB), then 2.7GB dropping to about 2GB and so on.


Re: Garbage collection

2020-06-27 Thread James Gray via Digitalmars-d-learn

On Saturday, 27 June 2020 at 12:53:01 UTC, James Gray wrote:
On Saturday, 27 June 2020 at 12:07:19 UTC, Stanislav Blinov 
wrote:

On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote:


[...]


That's a good observation. Although a GC implementation is not 
required to actually use malloc, so depending on that falls 
into "know what you're doing" territory :)


Thanks for the help, but unfortunately it isn't stopping memory 
usage growing in the original app. I will try and build a 
minimal example. In the meantime perhaps someone can suggest 
how I might figure out what is going on. Repeating the same 
action is giving memory usage growth as follows. 1.7GB first 
time (which now drops to about 1GB), then 2.7GB dropping to 
about 2GB and so on.


Which eventually results in mac os running out of memory.


Re: Garbage collection

2020-06-27 Thread Stanislav Blinov via Digitalmars-d-learn

On Saturday, 27 June 2020 at 11:35:12 UTC, Arafel wrote:

If you are using linux, have in mind that the memory is often 
not returned to the OS even after a (libc) free.


That's a good observation. Although a GC implementation is not 
required to actually use malloc, so depending on that falls into 
"know what you're doing" territory :)


Re: Garbage collection

2020-06-27 Thread Arafel via Digitalmars-d-learn

On 27/6/20 13:21, Stanislav Blinov wrote:


I would think collect + minimize should do the trick. Just keep in mind 
that that's grossly inefficient.


If you are using linux, have in mind that the memory is often not 
returned to the OS even after a (libc) free. If you check with tools 
like `top`, it'll still show as assigned to the process.


What I had to do (both in D and in C/C++) was to call malloc_trim [1] 
manually to have the memory actually sent back to the OS.


[1]: https://man7.org/linux/man-pages/man3/malloc_trim.3.html


Re: Garbage collection

2020-06-27 Thread Stanislav Blinov via Digitalmars-d-learn

On Saturday, 27 June 2020 at 11:11:38 UTC, James Gray wrote:


I am measuring the memory usage using top from the command line.
GC.minimize() does seem to stop the leak.


That is not a memory leak. That's the allocator keeping pages for 
itself to not have to go to the kernel every time you allocate.


But it doesn't explain why the program isn't releasing 
essentially all the memory between calls to f (it using around 
2GB ram all the time).


Allocators usually don't do that. They keep (at least some) 
memory mapped to make allocations more efficient.



Is there a way of achieving that?


I would think collect + minimize should do the trick. Just keep 
in mind that that's grossly inefficient.


Re: Garbage collection

2020-06-27 Thread Mike Parker via Digitalmars-d-learn

On Saturday, 27 June 2020 at 11:11:38 UTC, James Gray wrote:



I am measuring the memory usage using top from the command line.
GC.minimize() does seem to stop the leak. But it doesn't 
explain why
the program isn't releasing essentially all the memory between 
calls
to f (it using around 2GB ram all the time). Is there a way of 
achieving that?


It's not a leak. The GC allocates memory as it needs it and holds 
on to it. When something is collected, the GC can reuse then 
released memory when it needs it.


Re: Garbage collection

2020-06-27 Thread James Gray via Digitalmars-d-learn

On Saturday, 27 June 2020 at 11:00:58 UTC, Stanislav Blinov wrote:

On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:

I find that the memory usage grows to about 1.5GB and never 
decreases. Is there something I am not understanding?


How are you measuring that? GC.collect() does not necessarily 
release the pages to the OS. For that, there's the 
GC.minimize().


I am measuring the memory usage using top from the command line.
GC.minimize() does seem to stop the leak. But it doesn't explain 
why
the program isn't releasing essentially all the memory between 
calls
to f (it using around 2GB ram all the time). Is there a way of 
achieving that?


Re: Garbage collection

2020-06-27 Thread Stanislav Blinov via Digitalmars-d-learn

On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:

I find that the memory usage grows to about 1.5GB and never 
decreases. Is there something I am not understanding?


How are you measuring that? GC.collect() does not necessarily 
release the pages to the OS. For that, there's the GC.minimize().


Re: Garbage collection

2020-06-27 Thread James Gray via Digitalmars-d-learn

On Saturday, 27 June 2020 at 10:08:15 UTC, James Gray wrote:
I am writing a web application using vibe.d (not sure whether 
that is relevant or not), and have run into a memory leak. I 
wrote the following code to try and replicate the problem.


[...]


I now compiled the same code above with ldc2 and it is leaking.
Any suggestions?



Garbage collection

2020-06-27 Thread James Gray via Digitalmars-d-learn
I am writing a web application using vibe.d (not sure whether 
that is relevant or not), and have run into a memory leak. I 
wrote the following code to try and replicate the problem.


import std.algorithm;
import std.range;
import std.format;
import std.stdio;
import core.thread;
import core.memory;

auto f(R)(R r) {
 return format("%s", r);
}

void main()
{
 while(true)
 {
  writeln("Starting");
  {
   auto str = f(iota(1).map!(x=>x+1));
  }
  writeln("Done");
  GC.collect();
  Thread.sleep( dur!("msecs")( 3 ) );
 }
}

 It doesn't replicate the problem but it still doesn't behave as 
I would expect. I would expect the memory usage of this code to 
grow and shrink. However, I find that the memory usage grows to 
about 1.5GB and never decreases. Is there something I am not 
understanding?


Re: Garbage Collection Issue

2020-06-01 Thread IGotD- via Digitalmars-d-learn
On Monday, 1 June 2020 at 12:37:05 UTC, Steven Schveighoffer 
wrote:


I was under the impression that TLS works by altering a global 
pointer during the context switch. I didn't think accessing a 
variable involved a system call.


For sure they are slower than "normal" variables, but how much 
slower? I'm not sure.




It depends, there several different optimizations possible. This 
is essentially the difference between the -fPIC and -fpie flag 
GNU compilers. -fpie can optimize TLS so that it is an offset 
from a certain register (fs or gs with x86). Otherwise the 
compiler insert __tls_get_addr. Typically shared objects gets 
this call, but the executable can optimize. So if druntime is a 
shared object, it will use __tls_get_addr. TLS variables will not 
be major hit if used moderately, used in a loop, then you will 
certainly see a performance hit.




This can only take you so far, when the language uses TLS by 
default. The GC has to support scanning TLS and so it uses TLS 
to track thread-specific data.




Yes, this was some of the annoyance I had when porting druntime. 
The thread startup code needed to use link library (like 
elf/link.h for linux) in order to obtain the entire TLS area 
(areas because there a several of them). This includes scanning 
sections during startup and it becomes even more complicated with 
runtime loaded modules. Basically there is a lot of boiler plate 
in druntime just for reading the executable format. druntime has 
tons of elf stuff in it just to load a program something I'm not 
too keen on, because it's a lot of code and you need to support 
all the quirks with different CPU archs and operating systems. 
You'd want druntime to be more OS agnostic a let the OS services 
deal with the TLS stuff. The only upside can be that you can have 
a full symbolic stack trace during aborts when a poking in the 
executable formats.


Well, that's how it is because of GC and there is not really any 
way around it. A non tracing GC would not have this requirement 
though. When you dig into these details you realize how heavy the 
D language really is and some solutions get negative beauty 
points.










Re: Garbage Collection Issue

2020-06-01 Thread Steven Schveighoffer via Digitalmars-d-learn

On 6/1/20 6:51 AM, IGotD- wrote:

On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer wrote:


I can't imagine much of druntime working at all without TLS. Indeed, 
it is a requirement these days.


I believe that's where these roots are being stored.



I would really like if druntime could remove its TLS variables as much 
as possible. TLS is really a complicated solution underneath and 
druntime makes it even more complicated. It requires a hook in thread 
creation since the raw TLS specification only applies simple variables 
that can be initialized using memcpy/memset. Any thread that is created 
outside the druntime will fail if D supports "complex" TLS variables.


D can use non-D created threads, but they will not be scanned by the GC, 
or run thread static constructors or destructors.




TLS variables are also slower that normal variables since it often 
requires a system call in order to obtain the variable.


I was under the impression that TLS works by altering a global pointer 
during the context switch. I didn't think accessing a variable involved 
a system call.


For sure they are slower than "normal" variables, but how much slower? 
I'm not sure.




druntime should use stack variables much it can and/or shared variables.


druntime does not needlessly use TLS as far as I know. If you find a 
case that can be switched please file a bug report.




If you ever encounter a TLS variable which is global variable in D, try 
to see if you can solve it with a stack or shared variable.


This can only take you so far, when the language uses TLS by default. 
The GC has to support scanning TLS and so it uses TLS to track 
thread-specific data.


What it sounds like to me is that the OP implemented a "get it to 
compile" solution for TLS, and this is not working for him.


There is no removing TLS, because the language uses it directly for 
global variables, and many guarantees are enabled by it.


For instance the array append runtime uses a lock-free TLS cache to 
ensure speedy appending. Without TLS, the global lock would be needed 
for every append.


-Steve


Re: Garbage Collection Issue

2020-06-01 Thread Steven Schveighoffer via Digitalmars-d-learn

On 6/1/20 5:53 AM, a11e99z wrote:

On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer wrote:

I can't imagine much of druntime working at all without TLS. Indeed, 
it is a requirement these days.




TLS is evil for async/await when any thread can execute any fiber (case 
where fiber tied to thread is wrong/dead version of async/await cuz 1st 
thread has 1000 fibers and 2nd only 10)


That would require fiber local storage, which I don't know if that is 
supported.


I think fibers jumping between threads is also not supported, but I 
don't know for certain.


-Steve


Re: Garbage Collection Issue

2020-06-01 Thread IGotD- via Digitalmars-d-learn
On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer 
wrote:


I can't imagine much of druntime working at all without TLS. 
Indeed, it is a requirement these days.


I believe that's where these roots are being stored.

-Steve


I would really like if druntime could remove its TLS variables as 
much as possible. TLS is really a complicated solution underneath 
and druntime makes it even more complicated. It requires a hook 
in thread creation since the raw TLS specification only applies 
simple variables that can be initialized using memcpy/memset. Any 
thread that is created outside the druntime will fail if D 
supports "complex" TLS variables.


TLS variables are also slower that normal variables since it 
often requires a system call in order to obtain the variable.


druntime should use stack variables much it can and/or shared 
variables.


If you ever encounter a TLS variable which is global variable in 
D, try to see if you can solve it with a stack or shared variable.






Re: Garbage Collection Issue

2020-06-01 Thread a11e99z via Digitalmars-d-learn
On Sunday, 31 May 2020 at 16:57:06 UTC, Steven Schveighoffer 
wrote:


I can't imagine much of druntime working at all without TLS. 
Indeed, it is a requirement these days.




TLS is evil for async/await when any thread can execute any fiber 
(case where fiber tied to thread is wrong/dead version of 
async/await cuz 1st thread has 1000 fibers and 2nd only 10)


Re: Garbage Collection Issue

2020-05-31 Thread Steven Schveighoffer via Digitalmars-d-learn

On 5/30/20 9:51 PM, Marius Cristian Baciu wrote:

I am encountering a strange problem with the GC on a specific platform:
at the first attempt to clear the current memory pool to make room for a 
new allocation, the GC considers that the page in which the main thread 
resides (the one created in the init function of the GC) can be freed.. 
therefore, frees the entire pool and reallocates at the same location; 
later, when accessing thread's address, it stumbles upon garbage data.
The question is: where does the GC expects the address of the thread to 
be found so that it takes it into consideration?
A relevant mention would be that the platform doesn't support TLS so it 
won't find anything when trying to access that data. Could it be related 
to this?


I can't imagine much of druntime working at all without TLS. Indeed, it 
is a requirement these days.


I believe that's where these roots are being stored.

-Steve


Garbage Collection Issue

2020-05-30 Thread Marius Cristian Baciu via Digitalmars-d-learn
I am encountering a strange problem with the GC on a specific 
platform:
at the first attempt to clear the current memory pool to make 
room for a new allocation, the GC considers that the page in 
which the main thread resides (the one created in the init 
function of the GC) can be freed.. therefore, frees the entire 
pool and reallocates at the same location; later, when accessing 
thread's address, it stumbles upon garbage data.
The question is: where does the GC expects the address of the 
thread to be found so that it takes it into consideration?
A relevant mention would be that the platform doesn't support TLS 
so it won't find anything when trying to access that data. Could 
it be related to this?


Re: On D's garbage collection

2019-10-13 Thread Marcel via Digitalmars-d-learn
On Tuesday, 8 October 2019 at 16:48:55 UTC, Ferhat Kurtulmuş 
wrote:
On Tuesday, 8 October 2019 at 16:43:23 UTC, Ferhat Kurtulmuş 
wrote:

On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote:

[...]


I think you may find this interesting: 
https://www.auburnsounds.com/blog/2016-11-10_Running-D-without-its-runtime.html


And the code which is actually working: 
https://github.com/AuburnSounds/Dplug/blob/master/core/dplug/core/nogc.d


Wonderful, i'll be looking into it. Thank you!


Re: On D's garbage collection

2019-10-08 Thread Max Haughton via Digitalmars-d-learn

On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote:
I'm been thinking about using D in conjunction with C11 to 
develop a set of applications with hard real-time requirements. 
While initially the goal was to use C++ instead, it has become 
clear that D's introspection facilities will offer significant 
advantages. Unfortunately, the project will heavily rely on 
custom memory allocators written in C, so the presence of 
garbage collection in the language is a problem. While I'm 
aware that the nogc attribute exists, I haven't actually seen a 
way to apply it to a whole project. Is this possible?


Do you want to write D code that just doesn't use the GC or the 
whole runtime?


If the former then use @nogc at the entry point of your D code 
(This means that - say - main cannot call anything non-@nogc and 
therefore guarantees the program is @nogc), if the latter then 
use -betterC


IMO, if the interface to your memory allocators is stable then 
just link with them and write the whole thing in D (Interfacing 
with C is a solved problem but C is just awful compared to the 
features you get for free in D)


Re: On D's garbage collection

2019-10-08 Thread Ferhat Kurtulmuş via Digitalmars-d-learn
On Tuesday, 8 October 2019 at 16:43:23 UTC, Ferhat Kurtulmuş 
wrote:

On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote:
I'm been thinking about using D in conjunction with C11 to 
develop a set of applications with hard real-time 
requirements. While initially the goal was to use C++ instead, 
it has become clear that D's introspection facilities will 
offer significant advantages. Unfortunately, the project will 
heavily rely on custom memory allocators written in C, so the 
presence of garbage collection in the language is a problem. 
While I'm aware that the nogc attribute exists, I haven't 
actually seen a way to apply it to a whole project. Is this 
possible?


I think you may find this interesting: 
https://www.auburnsounds.com/blog/2016-11-10_Running-D-without-its-runtime.html


And the code which is actually working: 
https://github.com/AuburnSounds/Dplug/blob/master/core/dplug/core/nogc.d





Re: On D's garbage collection

2019-10-08 Thread Ferhat Kurtulmuş via Digitalmars-d-learn

On Tuesday, 8 October 2019 at 16:28:51 UTC, Marcel wrote:
I'm been thinking about using D in conjunction with C11 to 
develop a set of applications with hard real-time requirements. 
While initially the goal was to use C++ instead, it has become 
clear that D's introspection facilities will offer significant 
advantages. Unfortunately, the project will heavily rely on 
custom memory allocators written in C, so the presence of 
garbage collection in the language is a problem. While I'm 
aware that the nogc attribute exists, I haven't actually seen a 
way to apply it to a whole project. Is this possible?


I think you may find this interesting: 
https://www.auburnsounds.com/blog/2016-11-10_Running-D-without-its-runtime.html




On D's garbage collection

2019-10-08 Thread Marcel via Digitalmars-d-learn
I'm been thinking about using D in conjunction with C11 to 
develop a set of applications with hard real-time requirements. 
While initially the goal was to use C++ instead, it has become 
clear that D's introspection facilities will offer significant 
advantages. Unfortunately, the project will heavily rely on 
custom memory allocators written in C, so the presence of garbage 
collection in the language is a problem. While I'm aware that the 
nogc attribute exists, I haven't actually seen a way to apply it 
to a whole project. Is this possible?


Re: Memory reference that does not stop garbage collection.

2019-02-08 Thread Rene Zwanenburg via Digitalmars-d-learn

On Friday, 8 February 2019 at 15:42:13 UTC, Jonathan Levi wrote:

I should be able to use `core.memory.GC.removeRange` right?


That would leave dangling references in the array. You may be 
interested in this, but I have not used it myself:


https://repo.or.cz/w/iv.d.git/blob/HEAD:/weakref.d


Memory reference that does not stop garbage collection.

2019-02-08 Thread Jonathan Levi via Digitalmars-d-learn

I have observers and listeners.

class Observer {
   Listener[] listeners;
}
class Listener {}

The observers keep references to listeners, but I would like the 
GC to garbage collect listeners even if observers have references 
to it and remove the references in observers.


I should be able to use `core.memory.GC.removeRange` right?

class Observer {
  Listener[] listeners;
  this() {
listeners = [];
GC.removeRange(listeners.ptr);
  }
}

And in the deconstructor for listeners release the references in 
observers.
How does that work if the listeners array gets reallocated when 
extending, do I just recall `removeRange` any time after an 
append?


I am guessing I am not the first one to want this so I would 
rather not reinvent the wheel myself.  I also do not want memory 
leaks.


Thanks!


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-29 Thread Meta via Digitalmars-d-learn

On Wednesday, 28 June 2017 at 15:55:41 UTC, John Burton wrote:

On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
I'm coming from a C++ background so I'm not too used to 
garbage collection and it's implications. I have a function 
that creates a std.socket.Socket using new and connects to a 
tcp server, and writes some stuff to it. I then explicitly 
close the socket, and the socket object goes out of scope.




Am I doing this right? Or is there a better way to do this in 
D?


Thanks.



For my use case here, I'm increasingly thinking that just 
calling the underlying 'C' socket and send calls is better. No 
need for anything complicated at all for my actual program :)


One final piece of advice as this thread seemed to have gone off 
the rails a bit. You can always put a `scope(exit) 
socket.close();` after you create the socket. This will ensure 
that the socket will be closed once the scope is exited no matter 
what... almost, anyway. If an Error is thrown no stack unwinding 
is done but at this point your program is in an unrecoverable 
state anyway and you have a lot more to worry about than a socket 
that hasn't been closed.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread John Burton via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
I'm coming from a C++ background so I'm not too used to garbage 
collection and it's implications. I have a function that 
creates a std.socket.Socket using new and connects to a tcp 
server, and writes some stuff to it. I then explicitly close 
the socket, and the socket object goes out of scope.





Am I doing this right? Or is there a better way to do this in D?

Thanks.



For my use case here, I'm increasingly thinking that just calling 
the underlying 'C' socket and send calls is better. No need for 
anything complicated at all for my actual program :)




Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Mike Parker via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 13:19:37 UTC, Guillaume Piolat 
wrote:



https://forum.dlang.org/post/pmulowxpikjjffkrs...@forum.dlang.org

Not an issue with DerelictUtil, an issue with the strategy of 
closing resources in GC with this case.


Derelict loaders work-around this by not unloading shared 
libraries so the GC won't unload shared libs before the 
resources related to the shared library are freed.


https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366


Yeah, the loaders all used to needlessly unload the shared 
libraries in a static destructor. The fact that they don't 
anymore isn't to avoid any GC/destructor issues, but because 
there's no point in unloading the libraries when the system is 
going to do it anyway.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Moritz Maxeiner via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 12:33:24 UTC, Guillaume Piolat 
wrote:
On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner 
wrote:

Requirement: Do not allocate using the GC
Option 1) Use structs with `@disable this`, `@disable 
this(this)`, and a destructor that checks whether the resource 
reference is != invalid resource reference before trying to 
release.
Option 2) Use classes with simple constructor/destructor 
layout.
If you want to integrate a check that the requirement holds 
(with either class or struct), put `if (gc_inFinalizer) throw 
SomeError` into the class/struct destructor


I don't get it.


You asked for a "_simple_ story about resource release" and the 
above two options provide that IMHO.


It's completely possible to use the full power of GC and be 
deterministic.


Sure, but it's not "simple". With D's GC as the memory allocator 
you can currently have
(1) non-deterministic memory management and non-deterministic 
object lifetimes:
Use `new` or 
`std.experimental.allocator.make!(std.experimental.allocator.gc_allocator.GCAllocator)` for allocation
The GC will finalize objects with no pointers to them 
non-deterministically and deallocate the memory after their 
respective finalization.
(2) non-deterministic memory management and deterministic object 
lifetimes:
Construct the objects as with (1), but destruct them by 
calling `destroy` on them outside of a collection cycle (i.e. 
when they, and all their members are still considered "live" by 
the GC).
The GC will collect the memory non-deterministically after 
there are no more pointers to it.
Warning: You are responsible for ensuring that the GC will 
never see an undestroyed object in the collection cycle, because 
it might try to finalize (call its `~this`) it.
 You can protect yourself against such finalization 
attempts by putting `if (gc_inFinalizer) throw Error("Bug! I 
forgot to destruct!")` in the objects' respective destructors.
(3) deterministic memory management and deterministic object 
lifetimes:
Construct the objects as with (1), but destruct 
them by calling 
`std.experimental.allocator.dispose!(std.experimental.allocator.gc_allocator.GCAllocator)`on them outside of a collection cycle.

The GC will not do anything for those objects.
Same warning as (2) applies.


I'm out of this very _confused_ discussion.


I'm honestly not sure what's confusing about it.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Guillaume Piolat via Digitalmars-d-learn

On Wednesday, 28 June 2017 at 13:02:04 UTC, Mike Parker wrote:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:




So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.


What's the issue with DerelictUtil?


https://forum.dlang.org/post/pmulowxpikjjffkrs...@forum.dlang.org

Not an issue with DerelictUtil, an issue with the strategy of 
closing resources in GC with this case.


Derelict loaders work-around this by not unloading shared 
libraries so the GC won't unload shared libs before the resources 
related to the shared library are freed.


https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366




Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Mike Parker via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:




So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.


What's the issue with DerelictUtil?


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Moritz Maxeiner via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 12:28:28 UTC, Guillaume Piolat 
wrote:
On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner 
wrote:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:
So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.


I thought I had (implicitly): B needs to be `@disable 
finalize`.


So in the current language, doesn't exist?


I thought that was the premise of the discussion?
That the GC *currently* invokes the "destructors" as finalizers 
and that *currently* the only way to avoid that is calling 
`destroy` on them manually beforehand (since destructors won't be 
called twice on a single object).


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Guillaume Piolat via Digitalmars-d-learn

On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner wrote:

Requirement: Do not allocate using the GC
Option 1) Use structs with `@disable this`, `@disable 
this(this)`, and a destructor that checks whether the resource 
reference is != invalid resource reference before trying to 
release.

Option 2) Use classes with simple constructor/destructor layout.
If you want to integrate a check that the requirement holds 
(with either class or struct), put `if (gc_inFinalizer) throw 
SomeError` into the class/struct destructor


I don't get it.
It's completely possible to use the full power of GC and be 
deterministic.

I'm out of this very _confused_ discussion.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Guillaume Piolat via Digitalmars-d-learn

On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner wrote:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:
So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.


I thought I had (implicitly): B needs to be `@disable finalize`.


So in the current language, doesn't exist?


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Moritz Maxeiner via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 09:22:07 UTC, Guillaume Piolat 
wrote:
Deterministic destruction is a _solved_ problem in C++, and a 
number of users to convert are now coming from C++.


It is also in D, as long as you don't use the GC (which is 
inherently non-deterministic).


  3. Suggest a systematic, always working, workaround (or 
language change such as "GC doesn't call ~this). C++ users have 
no difficulty having an object graph with detailed ownership 
schemes, that's what they do day in day out.


It's important to have a _simple_ story about resource release, 
else we will talk about "GC" every day, and hearing about "GC" 
is bad for marketing.


Requirement: Do not allocate using the GC
Option 1) Use structs with `@disable this`, `@disable 
this(this)`, and a destructor that checks whether the resource 
reference is != invalid resource reference before trying to 
release.

Option 2) Use classes with simple constructor/destructor layout.
If you want to integrate a check that the requirement holds (with 
either class or struct), put `if (gc_inFinalizer) throw 
SomeError` into the class/struct destructor


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Moritz Maxeiner via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:
So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.


I thought I had (implicitly): B needs to be `@disable finalize`.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Guillaume Piolat via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
wrote:
There are definitely cases where finalizers make sense. Case 
in point: if you have a socket class, it makes perfect sense 
for it to have a finalizer. Yes, it's better to close it 
manually, but it will work just fine for the GC to close it 
when finalizing the class object so long as you don't use so 
many sockets that you run out before the GC collects them. So, 
having a finalizer is a good backup to ensure that the socket 
resource doesn't leak.


- Jonathan M Davis


So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.


And I'm gonna rant a little bit more.

Deterministic destruction is a _solved_ problem in C++, and a 
number of users to convert are now coming from C++.


We should:
  1. be honest and tell things as they are: it's more complicated 
than in C++, but also liberating when you know to juggle between 
GC and deterministic
  2. Avoid making bogus suggestions which don't always work, such 
as close() methods, releasing resource with GC, . They only work 
for _some_ resources.
  3. Suggest a systematic, always working, workaround (or 
language change such as "GC doesn't call ~this). C++ users have 
no difficulty having an object graph with detailed ownership 
schemes, that's what they do day in day out.


It's important to have a _simple_ story about resource release, 
else we will talk about "GC" every day, and hearing about "GC" is 
bad for marketing.





Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Guillaume Piolat via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
wrote:
There are definitely cases where finalizers make sense. Case in 
point: if you have a socket class, it makes perfect sense for 
it to have a finalizer. Yes, it's better to close it manually, 
but it will work just fine for the GC to close it when 
finalizing the class object so long as you don't use so many 
sockets that you run out before the GC collects them. So, 
having a finalizer is a good backup to ensure that the socket 
resource doesn't leak.


- Jonathan M Davis


So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.








Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Jacob Carlborg via Digitalmars-d-learn

On 2017-06-27 11:54, John Burton wrote:

I'm coming from a C++ background so I'm not too used to garbage
collection and it's implications. I have a function that creates a
std.socket.Socket using new and connects to a tcp server, and writes
some stuff to it. I then explicitly close the socket, and the socket
object goes out of scope.


Yes. You can use "scope (exit)", unless you're already doing so.


Now the issue is that I now need to call this function more than once
every second. I worry that it will create large amounts of uncollected
"garbage" which will eventually lead to problems.


Sounds like you need a connection pool. The vibe.d [1] framework 
contains connection pools.


It's also possible to manually disable the GC for a while and then 
enable it again.


[1] http://vibed.org

--
/Jacob Carlborg


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Jacob Carlborg via Digitalmars-d-learn

On 2017-06-27 17:24, Steven Schveighoffer wrote:


Yes, Tango solved this by having a separate "finalize()" method. I wish
we had something like this.


Not sure if this is the same, but I remember that Tango had a separate 
method called "dispose" that was called if a class was allocated on the 
stack, i.e. with the "scope" keyword.


--
/Jacob Carlborg


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-28 Thread Dsby via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
I'm coming from a C++ background so I'm not too used to garbage 
collection and it's implications. I have a function that 
creates a std.socket.Socket using new and connects to a tcp 
server, and writes some stuff to it. I then explicitly close 
the socket, and the socket object goes out of scope.


[...]


May be you can see yu: https://github.com/dushibaiyu/yu


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Moritz Maxeiner via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
wrote:
On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via 
Digitalmars-d-learn wrote:
Not every class can't be finalized, so it might make sense for 
finalization to remain an available option.


There are definitely cases where finalizers make sense. Case in 
point: if you have a socket class, it makes perfect sense for 
it to have a finalizer. Yes, it's better to close it manually, 
but it will work just fine for the GC to close it when 
finalizing the class object so long as you don't use so many 
sockets that you run out before the GC collects them. So, 
having a finalizer is a good backup to ensure that the socket 
resource doesn't leak.


Yes, I think that's like the File class I presented (at least on 
Posix).
But on the other hand, since there are also cases in which 
finalization of an alive (==undestroyed) object are unacceptable 
(e.g. when it requires other GC managed objects to be alive), 
splitting them off into two separate methods and allowing to 
forbid finalization seems sensible to me.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Jonathan M Davis via Digitalmars-d-learn
On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via Digitalmars-d-learn 
wrote:
> On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat
>
> wrote:
> > On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:
> >> - Replace calls by the GC to `~this` with calls to `finalize`
> >> (or invent some cool other shortened name for the latter)
> >
> > My point is that in such a "finalize()" function the only sane
> > things to do is to crash if the resource wasn't freed already.
> > Why so?
> >
> > [...]
>
> Not every class can't be finalized, so it might make sense for
> finalization to remain an available option.

There are definitely cases where finalizers make sense. Case in point: if
you have a socket class, it makes perfect sense for it to have a finalizer.
Yes, it's better to close it manually, but it will work just fine for the GC
to close it when finalizing the class object so long as you don't use so
many sockets that you run out before the GC collects them. So, having a
finalizer is a good backup to ensure that the socket resource doesn't leak.

- Jonathan M Davis


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Moritz Maxeiner via Digitalmars-d-learn
On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat 
wrote:

On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:
- Replace calls by the GC to `~this` with calls to `finalize` 
(or invent some cool other shortened name for the latter)


My point is that in such a "finalize()" function the only sane 
things to do is to crash if the resource wasn't freed already. 
Why so?


[...]


Not every class can't be finalized, so it might make sense for 
finalization to remain an available option.
What I think reasonable is treating destruction and finalization 
as two distinct things. This could look like the following:


---
// Does not refer to other GC memory, can be finalized
class File
{
private:
int fd;
public:
this() { fd = open(...); }
~this() { close(fd); }

// If the GC collects a File object that hasn't been 
destroyed yet, it will call this finalizer,
// which *manually* calls the destructor (because it's safe 
to do so in this case)
// A File object can still be manually `destroy`ed 
beforehand, in which case this particular finalizer is still 
safe, since an object won't be destroyed twice

finalize() { destroy(this); }
// Or maybe prettier: alias finalize = ~this;
}

class Foo
{
private:
File f;
public:
this() { f = new File(); }

// Make the process crash on finalization of an undestroyed 
Foo object

@disable finalize();
}
---


- Reserve `~this` for being called by deterministic lifetime 
management (std.experimental.allocator.dispose, 
object.destroy, RefCounted, Unique, etc.)


That's precisely what the GC-proof-resource-class allows, by 
preventing defective, non-composable usages of ~this.


The GC proof resource class idiom is a necessary hack, but it 
*is* a hack with its own limitations (e.g. depending on Error 
being catchable).


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Guillaume Piolat via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:


Do you mean destructors?


Yes.


- Replace calls by the GC to `~this` with calls to `finalize` 
(or invent some cool other shortened name for the latter)


My point is that in such a "finalize()" function the only sane 
things to do is to crash if the resource wasn't freed already. 
Why so?


See my texture example in: 
https://forum.dlang.org/post/kliasvljzwjhzuzib...@forum.dlang.org


More simply:
If A needs an alive B to be destroyed, then neither A-owns-B or 
B-owns-A can guarantee the ordering under GC destruction.


Furthermore, this is viral. Whoever owns A needs an alive B to be 
destroyed.




- Reserve `~this` for being called by deterministic lifetime 
management (std.experimental.allocator.dispose, object.destroy, 
RefCounted, Unique, etc.)


That's precisely what the GC-proof-resource-class allows, by 
preventing defective, non-composable usages of ~this.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Moritz Maxeiner via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 23:42:38 UTC, Guillaume Piolat wrote:

On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote:


Well, technically speaking the `~this` for D classes *is* a 
finalizer that you may optionally manually call (e.g. via 
destroy).
It would be nice, though, to change class `~this` into a 
destructor and move the finalization into an extra method like 
`finalize`. Write a DIP?


The only sane way out is to prevent the GC from calling 
constructors.


Do you mean destructors?
If so, that's what I meant with the "technically". If the may GC 
call it (automatically), it's a finalizer, not a destructor (in 
the usual sense, anyway).
- Replace calls by the GC to `~this` with calls to `finalize` (or 
invent some cool other shortened name for the latter)
- Reserve `~this` for being called by deterministic lifetime 
management (std.experimental.allocator.dispose, object.destroy, 
RefCounted, Unique, etc.)




Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Guillaume Piolat via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote:


Well, technically speaking the `~this` for D classes *is* a 
finalizer that you may optionally manually call (e.g. via 
destroy).
It would be nice, though, to change class `~this` into a 
destructor and move the finalization into an extra method like 
`finalize`. Write a DIP?


The only sane way out is to prevent the GC from calling 
constructors.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Guillaume Piolat via Digitalmars-d-learn
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer 
wrote:
The GC still needs to call something to clean up non-memory 
resources,


Well, I'd much prefer if the GC would simply not call 
destructors. Yes, destructor can work for _some_ resources, but 
because of ordering it also create accidental correctness, or 
just doesn't work with other resources.


but having a call to use when cleaning up deterministically can 
make resource management more efficient (even memory resource 
management).


Yes. My point is that the destructor is uniquely positionned to 
be that call.
(This would also enables back failing destructors through the use 
of exceptions.)


In current D the problem with allowing the GC to close resources 
is that there is no ordering of destructors. A sub-tree of the 
ownership graph is freed by the GC together, but sometime an 
order is required for releasing.


Typically: you want to release a SDL_Texture but the Derelict 
SharedLib object has been freed already, so you can't call 
sdl_releaseTexture. It doesn't matter that the texture object 
held a reference to the ShareLib object since the sub-tree is 
destroyed together. close() methods don't help with that, the 
problem is with the GC calling destructors.



And if _some_ resource needs an ordering, then this property leak 
on their owners, so little by little the whole ownership graph 
need determinism.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread bauss via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote:
On Tuesday, June 27, 2017 09:54:19 John Burton via 
Digitalmars-d-learn wrote:

[...]


Arguably, std.socket should have used structs instead of 
classes for sockets for precisely this reason (though there are 
some advantages in using inheritance with sockets). But yes, 
calling close manually is the correct thing to do. Relying on 
the GC to call a destructor/finalizer is error-prone. There is 
no guarantee that the memory will ever be freed (e.g. the 
runtime could choose to not bother doing cleanup on shutdown), 
and even if the GC does collect it, there are no guarantees 
about how soon it will do so. However, if you keep allocating 
memory with the GC, then over time, the GC will collect 
GC-allocated memory that isn't currently being used so that it 
can reuse the memory. So, you really don't need to worry about 
the memory unless it becomes a bottleneck. It will be collected 
and reused, not leaked.


[...]


I agree with that it should have been structs. The inheritance 
issue could be fixed by having a private member of the struct in 
a class, that way there could still be a class wrapper around it.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Steven Schveighoffer via Digitalmars-d-learn

On 6/27/17 2:04 PM, Moritz Maxeiner wrote:

On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer wrote:

On 6/27/17 9:25 AM, Guillaume Piolat wrote:

On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote:
But I would use a close method, and not destroy(obj). The reason is 
because often times, you have wrapper types around your socket type, 
and just one extra level of indirection means the destructor cannot 
be used to clean up the socket (you are not allowed to access 
GC-allocated resources in a destructor).


All destructor restrictions do not apply when it's not called by the GC.

There really are two categories of destructors: called by the GC and 
called deterministically. Their usage should not overlap.


Yes, Tango solved this by having a separate "finalize()" method. I 
wish we had something like this.


Well, technically speaking the `~this` for D classes *is* a finalizer 
that you may optionally manually call (e.g. via destroy).
It would be nice, though, to change class `~this` into a destructor and 
move the finalization into an extra method like `finalize`. Write a DIP?


I don't have enough motivation to do this. But if we do anything, we 
should look at what Tango has done. It may not work exactly for druntime 
because of existing code, but it obviously didn't need compiler changes 
to work.


-Steve


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Moritz Maxeiner via Digitalmars-d-learn
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer 
wrote:

On 6/27/17 9:25 AM, Guillaume Piolat wrote:
On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer 
wrote:
But I would use a close method, and not destroy(obj). The 
reason is because often times, you have wrapper types around 
your socket type, and just one extra level of indirection 
means the destructor cannot be used to clean up the socket 
(you are not allowed to access GC-allocated resources in a 
destructor).


All destructor restrictions do not apply when it's not called 
by the GC.


There really are two categories of destructors: called by the 
GC and called deterministically. Their usage should not 
overlap.


Yes, Tango solved this by having a separate "finalize()" 
method. I wish we had something like this.


Well, technically speaking the `~this` for D classes *is* a 
finalizer that you may optionally manually call (e.g. via 
destroy).
It would be nice, though, to change class `~this` into a 
destructor and move the finalization into an extra method like 
`finalize`. Write a DIP?


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Steven Schveighoffer via Digitalmars-d-learn

On 6/27/17 12:16 PM, Steven Schveighoffer wrote:

On 6/27/17 11:24 AM, Steven Schveighoffer wrote:

On 6/27/17 9:25 AM, Guillaume Piolat wrote:


That's how the GC-proof resource class came to existence, after many 
destruction bugs, and it let's you use the GC as a detector for 
non-deterministic destruction. I miss it in @nogc :)


https://p0nce.github.io/d-idioms/#GC-proof-resource-class


There are definitely better ways to do this than trying to allocate 
and catching an error. The GC knows the GC is running, use that 
mechanism instead :)


https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189
https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150

Apparently that info is limited to the core.memory package. But it's 
extern(C), so declaring the prototype should work. Should be much more 
efficient than allocating a byte (which I see you don't delete if it 
works), and catching an error.


Just added this enhancement to publicize this function: 
https://issues.dlang.org/show_bug.cgi?id=17563


-Steve


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Steven Schveighoffer via Digitalmars-d-learn

On 6/27/17 11:24 AM, Steven Schveighoffer wrote:

On 6/27/17 9:25 AM, Guillaume Piolat wrote:


That's how the GC-proof resource class came to existence, after many 
destruction bugs, and it let's you use the GC as a detector for 
non-deterministic destruction. I miss it in @nogc :)


https://p0nce.github.io/d-idioms/#GC-proof-resource-class


There are definitely better ways to do this than trying to allocate and 
catching an error. The GC knows the GC is running, use that mechanism 
instead :)


https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189
https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150

Apparently that info is limited to the core.memory package. But it's 
extern(C), so declaring the prototype should work. Should be much more 
efficient than allocating a byte (which I see you don't delete if it 
works), and catching an error.


-Steve


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Steven Schveighoffer via Digitalmars-d-learn

On 6/27/17 9:25 AM, Guillaume Piolat wrote:

On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote:
But I would use a close method, and not destroy(obj). The reason is 
because often times, you have wrapper types around your socket type, 
and just one extra level of indirection means the destructor cannot be 
used to clean up the socket (you are not allowed to access 
GC-allocated resources in a destructor).


All destructor restrictions do not apply when it's not called by the GC.

There really are two categories of destructors: called by the GC and 
called deterministically. Their usage should not overlap.


Yes, Tango solved this by having a separate "finalize()" method. I wish 
we had something like this.




My reasoning went with the following:

1 - "I should have close() methods so that I don't rely on the GC, and 
the GC is calling ~this from the wrong thread etc, not everything can be 
released in this context (eg: OpenGL objects should be released from one 
thread only). Close methods will call close methods of "owned" objects."


I hadn't realized this, that is a good point. However, not all resources 
are restricted this much.


That's how the GC-proof resource class came to existence, after many 
destruction bugs, and it let's you use the GC as a detector for 
non-deterministic destruction. I miss it in @nogc :)


https://p0nce.github.io/d-idioms/#GC-proof-resource-class


There are definitely better ways to do this than trying to allocate and 
catching an error. The GC knows the GC is running, use that mechanism 
instead :)


Remember years ago when Alexandrescu suggested the GC shouldn't call 
heap destructors? That's what we get for having said no at the time.


No, the issue is that there isn't a separate call for destruction and 
GC. I think we should add this.


The GC still needs to call something to clean up non-memory resources, 
but having a call to use when cleaning up deterministically can make 
resource management more efficient (even memory resource management).


-Steve

-Steve


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Guillaume Piolat via Digitalmars-d-learn
On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer 
wrote:
But I would use a close method, and not destroy(obj). The 
reason is because often times, you have wrapper types around 
your socket type, and just one extra level of indirection means 
the destructor cannot be used to clean up the socket (you are 
not allowed to access GC-allocated resources in a destructor).


All destructor restrictions do not apply when it's not called by 
the GC.


There really are two categories of destructors: called by the GC 
and called deterministically. Their usage should not overlap.





My reasoning went with the following:

1 - "I should have close() methods so that I don't rely on the 
GC, and the GC is calling ~this from the wrong thread etc, not 
everything can be released in this context (eg: OpenGL objects 
should be released from one thread only). Close methods will call 
close methods of "owned" objects."


2 - "Uh, oh. Refcounted and Unique and Scoped all use .destroy, 
so destructors should call close() methods. To avoid double 
release, close should handle being called several times. The GC 
will close what I forgot!"


3 - "Actually there is no order of destruction when called by the 
GC. So I can't release owned objects when called by the GC. 
Better do nothing when close() is called by the GC, let's detect 
this.


4 - close() is now identical with ~this (at least for classes). 
Let's remove the close() method. Crash when a resource is freed 
by the GC.


That's how the GC-proof resource class came to existence, after 
many destruction bugs, and it let's you use the GC as a detector 
for non-deterministic destruction. I miss it in @nogc :)


https://p0nce.github.io/d-idioms/#GC-proof-resource-class

Remember years ago when Alexandrescu suggested the GC shouldn't 
call heap destructors? That's what we get for having said no at 
the time.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Steven Schveighoffer via Digitalmars-d-learn

On 6/27/17 8:29 AM, Guillaume Piolat wrote:

On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
Am I doing this right with GC? In C++ I'd ensure that the Socket class 
had a destructor that closed the socket and I'd also assume that once 
it went out of scope there was no memory left allocated. In D am I 
right to assume I need to manually close the socket but there is no 
need to worry about the memory?


Yes.
You can also call a destructor manually with destroy(obj);
This avoids having a forest of 'close' methods, who are duplicates of 
the destructor in spirit, and let's you use destructors like you are 
accustomed in C++.


Generally, it is dangerous to let the GC handle resource release:
https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors


This is the best to read, it's very tricky for people coming from other 
languages I think to understand the rules around the GC destructor.


I think you have gotten sound advice from many people, but I wanted to 
chime in as someone who used the GC to clean up sockets and file handles 
in a long-running program. Even with all these warnings, it does seem to 
work. My advice is to close the socket in the destructor if still valid, 
and then manually close it also. This way you don't leak resources if 
you miss a close call.


The fundamental problem with closing sockets and other resources with 
the GC is that the resource management issues and rules for memory are 
vastly different than those of other resources. The OS can give you vast 
sums of memory, but will likely limit your ability to keep sockets open. 
It also keeps the other end potentially locked up. If you try to 
allocate a new socket and run out of sockets, there is no "socket 
collection cycle". This is why closing manually is advised.


But I would use a close method, and not destroy(obj). The reason is 
because often times, you have wrapper types around your socket type, and 
just one extra level of indirection means the destructor cannot be used 
to clean up the socket (you are not allowed to access GC-allocated 
resources in a destructor).


Using std.typecons.RefCounted also can help to ensure destruction of a 
socket when it's no longer used. But like I said, it's not always 
possible to use a destructor of a higher level object to clean up a socket.


-Steve


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread cym13 via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 12:29:03 UTC, Guillaume Piolat wrote:

[...]


Hmm... Isn't it possible to just allocate the object/a pool of 
objects outside the loop and reuse it? Everybody's proposing 
other means of allocations which is nice, but I wonder why there 
is such a need for reallocations in the first place.


Of course the specifics depend on the implementation, maybe it 
isn't as straightforward. It's just that no allocation is faster 
than reallocation no matter what reallocation method is used.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Guillaume Piolat via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
I assume that I do need to explicitly call close on the socket 
as there is no deterministic destructor for class objects.


Yes.


I further assume that the runtime will garbage collect any 
memory allocated to the socket object at a later time.


Most probably.



Am I doing this right with GC? In C++ I'd ensure that the 
Socket class had a destructor that closed the socket and I'd 
also assume that once it went out of scope there was no memory 
left allocated. In D am I right to assume I need to manually 
close the socket but there is no need to worry about the memory?


Yes.
You can also call a destructor manually with destroy(obj);
This avoids having a forest of 'close' methods, who are 
duplicates of the destructor in spirit, and let's you use 
destructors like you are accustomed in C++.


Generally, it is dangerous to let the GC handle resource release:
https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors

Now the issue is that I now need to call this function more 
than once every second. I worry that it will create large 
amounts of uncollected "garbage" which will eventually lead to 
problems.


If this is a problem you can create it on the stack with 
std.typecons.scoped




Am I doing this right? Or is there a better way to do this in D?



What you are doing it OK.




Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Moritz Maxeiner via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
Now the issue is that I now need to call this function more 
than once every second. I worry that it will create large 
amounts of uncollected "garbage" which will eventually lead to 
problems.


Since nobody has mentioned Allocator, yet:
As you seem to know the lifetime of the socket statically, you 
can just use std.experimental.allocator.{make,dispose} [1]. With 
regards to reusing the memory: Simply use a freelist allocator 
[2].


[1] https://dlang.org/phobos/std_experimental_allocator.html#.make
[2] 
https://dlang.org/phobos/std_experimental_allocator_building_blocks_free_list.html


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Nicholas Wilson via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 11:43:27 UTC, John Burton wrote:
On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis 
wrote:
On Tuesday, June 27, 2017 09:54:19 John Burton via 
Digitalmars-d-learn wrote:
I'm coming from a C++ background so I'm not too used to 
garbage collection and it's implications. I have a function 
that creates a std.socket.Socket using new and connects to a 
tcp server, and writes some stuff to it. I then explicitly 
close the socket, and the socket object goes out of scope.


Thank you for the advice everyone.

The hardest part about learning D isn't the language, or how to 
program, it's unlearning what you know from C++ and learning 
the proper way to do things in D. I've tried D several times 
before and eventually stopped when I get to the stage of "how 
do I do this c++ thing in d" proves to be hard.


Instead this time, I've started writing D programs as "better 
C" and then slowly started adding in higher level d features. 
It's going much better as I'm no longer trying so hard to write 
bad C++ in D :)


Heh, there's no reason you can't, you'll just cause a few head 
scratches when you post code while we figure out what it does. 
Idiomatic D is rather different to C++ and is centered around 
ranges + algorithms, but you can write code in pretty much any 
style. Some of the advanced metaprogramming takes a bit to get 
your head around.


If you do get stuck do post a question here, we're more than 
happy to help.


Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread John Burton via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote:
On Tuesday, June 27, 2017 09:54:19 John Burton via 
Digitalmars-d-learn wrote:
I'm coming from a C++ background so I'm not too used to 
garbage collection and it's implications. I have a function 
that creates a std.socket.Socket using new and connects to a 
tcp server, and writes some stuff to it. I then explicitly 
close the socket, and the socket object goes out of scope.


Thank you for the advice everyone.

The hardest part about learning D isn't the language, or how to 
program, it's unlearning what you know from C++ and learning the 
proper way to do things in D. I've tried D several times before 
and eventually stopped when I get to the stage of "how do I do 
this c++ thing in d" proves to be hard.


Instead this time, I've started writing D programs as "better C" 
and then slowly started adding in higher level d features. It's 
going much better as I'm no longer trying so hard to write bad 
C++ in D :)






Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Jonathan M Davis via Digitalmars-d-learn
On Tuesday, June 27, 2017 09:54:19 John Burton via Digitalmars-d-learn 
wrote:
> I'm coming from a C++ background so I'm not too used to garbage
> collection and it's implications. I have a function that creates
> a std.socket.Socket using new and connects to a tcp server, and
> writes some stuff to it. I then explicitly close the socket, and
> the socket object goes out of scope.
>
> I assume that I do need to explicitly call close on the socket as
> there is no deterministic destructor for class objects. I further
> assume that the runtime will garbage collect any memory allocated
> to the socket object at a later time.
>
> Am I doing this right with GC? In C++ I'd ensure that the Socket
> class had a destructor that closed the socket and I'd also assume
> that once it went out of scope there was no memory left
> allocated. In D am I right to assume I need to manually close the
> socket but there is no need to worry about the memory?
>
> Now the issue is that I now need to call this function more than
> once every second. I worry that it will create large amounts of
> uncollected "garbage" which will eventually lead to problems.
>
> Am I doing this right? Or is there a better way to do this in D?

Arguably, std.socket should have used structs instead of classes for sockets
for precisely this reason (though there are some advantages in using
inheritance with sockets). But yes, calling close manually is the correct
thing to do. Relying on the GC to call a destructor/finalizer is
error-prone. There is no guarantee that the memory will ever be freed (e.g.
the runtime could choose to not bother doing cleanup on shutdown), and even
if the GC does collect it, there are no guarantees about how soon it will do
so. However, if you keep allocating memory with the GC, then over time, the
GC will collect GC-allocated memory that isn't currently being used so that
it can reuse the memory. So, you really don't need to worry about the memory
unless it becomes a bottleneck. It will be collected and reused, not leaked.

And if in C++, you would be newing up a socket object each time and then
deleting it rather than having the socket on the stack, then that
performance could actually be worse than using the GC and having it collect
the memory when it determines that it needs more memory.

If you do find that it becomes a bottleneck to be allocating all of these
sockets, then you can look into using std.typecons.scoped, which would
allocate a class on the stack, but that really only works if you're just
dealing with a local variable, and in general, I wouldn't worry about
anything like that unless the heap allocations are proving to be a
performance problem - particularly because you need to be very careful when
using it to avoid screwing up and having memory corruption, because you used
an object after it was freed.

- Jonathan M Davis



Re: Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread Nicholas Wilson via Digitalmars-d-learn

On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
I'm coming from a C++ background so I'm not too used to garbage 
collection and it's implications. I have a function that 
creates a std.socket.Socket using new and connects to a tcp 
server, and writes some stuff to it. I then explicitly close 
the socket, and the socket object goes out of scope.


I assume that I do need to explicitly call close on the socket 
as there is no deterministic destructor for class objects. I 
further assume that the runtime will garbage collect any memory 
allocated to the socket object at a later time.


Am I doing this right with GC? In C++ I'd ensure that the 
Socket class had a destructor that closed the socket and I'd 
also assume that once it went out of scope there was no memory 
left allocated. In D am I right to assume I need to manually 
close the socket but there is no need to worry about the memory?


Yes. from the documentation

Immediately drop any connections and release socket resources. 
Calling shutdown before close is recommended for 
connection-oriented sockets. The Socket object is no longer 
usable after close. Calling shutdown() before this is 
recommended for connection-oriented sockets.


The GC (if you use it) will pick up the garbage for you.

Now the issue is that I now need to call this function more 
than once every second. I worry that it will create large 
amounts of uncollected "garbage" which will eventually lead to 
problems.


Have a look at automem (https://github.com/atilaneves/automem/)



Am I doing this right? Or is there a better way to do this in D?

Thanks.


There is also vibe.d if you are looking for more networking stuff.
https://github.com/rejectedsoftware/vibe.d

Best of luck
Nic



Advice wanted on garbage collection of sockets for c++ programmer using D

2017-06-27 Thread John Burton via Digitalmars-d-learn
I'm coming from a C++ background so I'm not too used to garbage 
collection and it's implications. I have a function that creates 
a std.socket.Socket using new and connects to a tcp server, and 
writes some stuff to it. I then explicitly close the socket, and 
the socket object goes out of scope.


I assume that I do need to explicitly call close on the socket as 
there is no deterministic destructor for class objects. I further 
assume that the runtime will garbage collect any memory allocated 
to the socket object at a later time.


Am I doing this right with GC? In C++ I'd ensure that the Socket 
class had a destructor that closed the socket and I'd also assume 
that once it went out of scope there was no memory left 
allocated. In D am I right to assume I need to manually close the 
socket but there is no need to worry about the memory?


Now the issue is that I now need to call this function more than 
once every second. I worry that it will create large amounts of 
uncollected "garbage" which will eventually lead to problems.


Am I doing this right? Or is there a better way to do this in D?

Thanks.


Re: Garbage collection and closures.

2017-06-19 Thread Adam D. Ruppe via Digitalmars-d-learn

On Monday, 19 June 2017 at 09:18:56 UTC, Dsby wrote:

void uses(scope void delegate() dg);

will it be not alloc memory?


I test it , if use scope it will not alloc memony.


Right, using `scope` at the point the delegate variable is 
defined means it will never allocate.


Re: Garbage collection and closures.

2017-06-19 Thread Dsby via Digitalmars-d-learn

On Monday, 19 June 2017 at 09:10:16 UTC, Dsby wrote:

On Saturday, 17 June 2017 at 17:15:50 UTC, Adam D. Ruppe wrote:

On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote:

[...]


Where the variable is defined that is referenced in the 
closure.


So:

[...]


if the uses parma is 'scope':

void uses(scope void delegate() dg);

will it be not alloc memory?


I test it , if use scope it will not alloc memony.

import std.stdio;

void call(void delegate() fun)
{
fun();
}

void call2(scope void delegate() fun)
{
fun();
}

void main()
{
int a = 10;
   // call((){writeln(a);});
call2((){writeln(a);});
}
dmd -vgc ./t.d
it will not print anything.

if use call:
void main()
{
int a = 10;
call((){writeln(a);});
   // call2((){writeln(a);});
}

dmd -vgc ./t.d
  
 182ms  2017年06月19日 星期一 17时16分47秒

./t.d(13): vgc: using closure causes GC allocation



Re: Garbage collection and closures.

2017-06-19 Thread Dsby via Digitalmars-d-learn

On Saturday, 17 June 2017 at 17:15:50 UTC, Adam D. Ruppe wrote:

On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote:

[...]


Where the variable is defined that is referenced in the closure.

So:

[...]


if the uses parma is 'scope':

void uses(scope void delegate() dg);

will it be not alloc memory?





Re: Garbage collection and closures.

2017-06-17 Thread ANtlord via Digitalmars-d-learn

On Saturday, 17 June 2017 at 17:15:50 UTC, Adam D. Ruppe wrote:

On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote:
Excuse me, I can't get what does it mean "deepest-referenced". 
What the deep you mean? The deep of a closure or deep of the 
function where the variable is defined. Can you give an 
example code?


Where the variable is defined that is referenced in the closure.

So:

---
void uses(void delegate() dg);

void foo() {
   int a;
   foreach(b; 0 .. 10) {
  uses( () { a++; } ); // #1
  uses( () { b++; } ); // #2
   }
}
---


In that case, #1 would only be allocated once, at the start of 
the foo() function. It only uses `a`, so it doesn't have to 
allocate again after the point a is defined.


But #2 might allocate each time through the loop. (It currently 
doesn't, but this is filed as an open bug because it is 
supposed to.) Since it uses `b` which is defined inside the 
loop, it will have to allocate a new copy for each iteration.


Is this function called every time when allocation happens in 
a heap?


Not any allocation, it is just the function the compiler uses 
when it needs to make a closure.


Thanks a lot, Adam! Everything is clear. Except for the bug. I've 
got an expected result of a value of the variable from #2. The 
value equals to number from sequence plus one in each iteration. 
There are ten iterations.


What's wrong? Are there should be five iterations? It doesn't 
make sense for me due to the variable assigned value from the 
sequence 0..10. Or do I look at the wrong place? Can you give me 
a link to the bug?


Thank you again!


Re: Garbage collection and closures.

2017-06-17 Thread Adam D. Ruppe via Digitalmars-d-learn

On Saturday, 17 June 2017 at 14:19:34 UTC, ANtlord wrote:
Excuse me, I can't get what does it mean "deepest-referenced". 
What the deep you mean? The deep of a closure or deep of the 
function where the variable is defined. Can you give an example 
code?


Where the variable is defined that is referenced in the closure.

So:

---
void uses(void delegate() dg);

void foo() {
   int a;
   foreach(b; 0 .. 10) {
  uses( () { a++; } ); // #1
  uses( () { b++; } ); // #2
   }
}
---


In that case, #1 would only be allocated once, at the start of 
the foo() function. It only uses `a`, so it doesn't have to 
allocate again after the point a is defined.


But #2 might allocate each time through the loop. (It currently 
doesn't, but this is filed as an open bug because it is supposed 
to.) Since it uses `b` which is defined inside the loop, it will 
have to allocate a new copy for each iteration.


Is this function called every time when allocation happens in a 
heap?


Not any allocation, it is just the function the compiler uses 
when it needs to make a closure.


Re: Garbage collection and closures.

2017-06-17 Thread ANtlord via Digitalmars-d-learn

On Saturday, 17 June 2017 at 13:13:17 UTC, Adam D. Ruppe wrote:

On Saturday, 17 June 2017 at 13:03:28 UTC, ANtlord wrote:

Is GC called every iteration of this loop?


No, it will once on scope entry; where the deepest-referenced 
variable that is actually captured is defined. The compiler 
allocates heap space instead of stack space for the locals, 
then runs the function normally using that space.


Excuse me, I can't get what does it mean "deepest-referenced". 
What the deep you mean? The deep of a closure or deep of the 
function where the variable is defined. Can you give an example 
code?


It will only alloc once. You can prove this with a debugger 
btw, set a breakpoint on `_d_allocmemory`.


Is this function called every time when allocation happens in a 
heap?


Thank you. Sorry if my English is not clear.


Re: Garbage collection and closures.

2017-06-17 Thread Adam D. Ruppe via Digitalmars-d-learn

On Saturday, 17 June 2017 at 13:03:28 UTC, ANtlord wrote:

Is GC called every iteration of this loop?


No, it will once on scope entry; where the deepest-referenced 
variable that is actually captured is defined. The compiler 
allocates heap space instead of stack space for the locals, then 
runs the function normally using that space.


The current implementation doesn't quite do this - it actually 
will alloc only once, on function entry, even when there's a 
variable inside a loop that is supposed to be independent. This 
is the cause of the commonly-referenced bug with foreach(i; 
whatever) capture(i); all showing the same number.


When that bug is fixed, it is possible to allocate on each 
loop... but your code doesn't anyway, so you're ok.


It will only alloc once. You can prove this with a debugger btw, 
set a breakpoint on `_d_allocmemory`.


Garbage collection and closures.

2017-06-17 Thread ANtlord via Digitalmars-d-learn
Hello! I can't understand one thing related to closures and 
calling of GC. I have the following demo snippet, where a closure 
is passed to `receive` function in a loop.


bool isDone = false;
while(!isDone)
receive((bool val){ isDone = val });

Is GC called every iteration of this loop?

I know that I can move the code to a method of a struct and make 
the closure and variable "isDone" as fields of the struct. It 
avoids GC calling definitely. But I want to get how many times GC 
is called in case of a closure in a loop.


Thanks.


Re: reference to delegates and garbage collection

2014-07-15 Thread Kagamin via Digitalmars-d-learn
Since you access a field through `a` instance, this is usually 
done by loading the instance address into some register and 
reading from a location at a certain offset from that register


mov esi, [ebp-4] # the instance address
mov eax, [esi+8] # first field
...
mov [ebp-4], 0 # clear stack variable
...
call collect
...
ret # from main


collect:
...
push esi
...
pop esi
ret


Note the instance address is preserved in a register even if it's 
not needed. And this happen only if you read a field.


reference to delegates and garbage collection

2014-07-14 Thread AnFive via Digitalmars-d-learn

Hello everybody. I am new to D, and I am trying to familiarize
with all the new (to me) features.
I have a question about a strange behavior I cannot understand.
(In case it matters, I am using gdc 2.065 for Windows, but the
same happens with dmd).

Example code :

class A {   
void delegate() del;

~this() { writeln(A Destructor called); }   
}

class B {   
void fun() {}   
}

A makeA() { 
auto a = new A();
a.del = (new B().fun);
return a;
}

void main() {

auto a = makeA();
 /* Magic line goes here, see below */
a = null;

foreach(i; 0..10) {
writeln(Collecting);
GC.collect();   
}

writeln(End);   
}

If I run this code, the instance of A is of course collected at
the first call to GC.collect().
But if I call the delegate (i.e. a.del() ) in the magic line,
the instance is not destroyed until the end of the main. If I
wrap the call to a.del() in another function or method, the
instance is collected.

Can somebody explain why this happens? Is it just a strangeness
of the GC? Does calling the delegate keep a reference to a in the
current scope? Am I going crazy? I spent about half an hour
looking for a non-existent memory leak in my program because of
this behavior, so I'd like to have an explanation.

Thanks in advance!


Re: reference to delegates and garbage collection

2014-07-14 Thread Adam D. Ruppe via Digitalmars-d-learn
I'm just guessing, but it looks to me that the delegate's pointer 
might be on the stack there and isn't overwritten when the one 
function returns, so the gc still thinks there might be an active 
pointer to it.


Temporarily protect array from garbage collection

2014-04-24 Thread Lars T. Kyllingstad via Digitalmars-d-learn
Is it possible to temporarily prevent the garbage collector from 
collecting a memory block even if there are no references to it?


The use case is as follows:  I want to call a C library function 
which expects to take ownership of a buffer.  It looks something 
like this:


alias FreeFunc = extern(C) void function(void*, void*) 
nothrow;


extern(C) void foo(void* buf, size_t len,
   FreeFunc free, void* ctx) nothrow;

Here, 'buf' is a pointer to the buffer, 'len' is the length of 
the buffer, 'free' is a function to deallocate the buffer when 
the library is done with it, and 'ctx' is a user-supplied context 
pointer.  Upon deallocation, 'free' receives two parameters; the 
pointer to the buffer and the context pointer.  The latter can be 
anything, even null, as it is just passed to 'free' and not used 
for anything else.


Here is the problem:  I want to be able to use a 
garbage-collected dynamic array with this function, but I don't 
want to have to retain a reference to it in my program.  (I don't 
know when the C library will call the free function.)  In other 
words, I want something like this:


extern(C) void myFree(void* ptr, void* ctx)
{
enableGCFor(ptr);
}

auto arr = new int[123];
disableGCFor(arr);
foo(arr.ptr, arr.length, myFree, null);
arr = null;

Is this at all possible?

Thanks,
Lars


Re: Temporarily protect array from garbage collection

2014-04-24 Thread Justin Whear via Digitalmars-d-learn
On Thu, 24 Apr 2014 19:55:37 +, Lars T. Kyllingstad wrote:

 Is it possible to temporarily prevent the garbage collector from
 collecting a memory block even if there are no references to it?
 
 The use case is as follows:  I want to call a C library function which
 expects to take ownership of a buffer.  It looks something like this:
 
  alias FreeFunc = extern(C) void function(void*, void*)
 nothrow;
 
  extern(C) void foo(void* buf, size_t len,
 FreeFunc free, void* ctx) nothrow;
 
 Here, 'buf' is a pointer to the buffer, 'len' is the length of the
 buffer, 'free' is a function to deallocate the buffer when the library
 is done with it, and 'ctx' is a user-supplied context pointer.  Upon
 deallocation, 'free' receives two parameters; the pointer to the buffer
 and the context pointer.  The latter can be anything, even null, as it
 is just passed to 'free' and not used for anything else.
 
 Here is the problem:  I want to be able to use a garbage-collected
 dynamic array with this function, but I don't want to have to retain a
 reference to it in my program.  (I don't know when the C library will
 call the free function.)  In other words, I want something like this:
 
  extern(C) void myFree(void* ptr, void* ctx)
  {
  enableGCFor(ptr);
  }
 
  auto arr = new int[123];
  disableGCFor(arr);
  foo(arr.ptr, arr.length, myFree, null);
  arr = null;
 
 Is this at all possible?
 
 Thanks,
 Lars

You can use GC.addRoot() from core.memory before passing the pointer to 
the C function, then use GC.removeRoot in your myFree function.


Re: Temporarily protect array from garbage collection

2014-04-24 Thread Lars T. Kyllingstad via Digitalmars-d-learn

On Thursday, 24 April 2014 at 20:09:38 UTC, Justin Whear wrote:
You can use GC.addRoot() from core.memory before passing the 
pointer to

the C function, then use GC.removeRoot in your myFree function.


Perfect, thanks!


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-11 Thread Jacob Carlborg

On 2013-07-11 04:59, Maxim Fomin wrote:


To be pedantic dynamic arrays are implemented in D simply as

struct Array
{
 size_t length;
 void* ptr;
}

and there is no type parametrization since such arrays handling is
opaque for users (in druntime they are treated as void[]).
Parametrization can be useful in user side since performing any
operations with structure above (void*) will lead to errors. But in user
side there is no point in manipulating the structure directly, as it can
be done using usual properties/druntime without template bloat. By the
way, the style ABI page is written, allows implementation to have more
fields.


typeof(asd.ptr) gives back immutable(char)*, not immutable(void)*. So 
from a user point of view it's as if the array is templated.


--
/Jacob Carlborg


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-11 Thread Jacob Carlborg

On 2013-07-10 20:22, Ali Çehreli wrote:


And to be pedantic, length comes first:

struct Array (T)
{
 size_t length;
 T* ptr;
}


I thought ptr came first, that's the reason you could cast to the 
pointer type. Not that one should do that. Perhaps there's some 
compiler/runtime magic involved.


--
/Jacob Carlborg


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-11 Thread Jacob Carlborg

On 2013-07-11 09:43, Maxim Fomin wrote:


It's in the user side. In druntime it is void[] + typeinfo. I am not
aware of any part in dmd/druntime where arrays are repsented as
templates (or strongly typed) as depicted in this dicsussion. And
current treatment can be barely called templatization as there is no
templates, template instantiations and typicall horrible mangling at
all. More precise description is not templatization but some kind of
implicit conversion from array of specific type in source code to void
array plus typeinfo in runtime library. If user tries to use struct
Array(T) {...} instead of usual arrays he will gain no benefit but
useless template bloat.


Yes, but that is easier to type. All the above or:

struct Array (T)
{
size_t length;
T* ptr;
}

--
/Jacob Carlborg


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-11 Thread Jacob Carlborg

On 2013-07-11 13:19, Jacob Carlborg wrote:


Yes, but that is easier to type. All the above or:

struct Array (T)
{
 size_t length;
 T* ptr;
}



that should have been what.

--
/Jacob Carlborg


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-11 Thread Ali Çehreli

On 07/11/2013 12:23 AM, Jacob Carlborg wrote:

 On 2013-07-10 20:22, Ali Çehreli wrote:

 And to be pedantic, length comes first:

 struct Array (T)
 {
  size_t length;
  T* ptr;
 }

 I thought ptr came first, that's the reason you could cast to the
 pointer type. Not that one should do that. Perhaps there's some
 compiler/runtime magic involved.

There must be little magic and that magic should be the same as getting 
the .ptr property. Otherwise, the value of a struct object cannot be 
casted to pointer type:


struct S
{
int *p;
}

auto s = S();
int *p = cast(int*)s;

Error: e2ir: cannot cast s of type S to type int*

Ali



Re: pointers, assignments, Garbage Collection Oh My?

2013-07-11 Thread Maxim Fomin

On Thursday, 11 July 2013 at 17:07:18 UTC, Ali Çehreli wrote:

On 07/11/2013 12:23 AM, Jacob Carlborg wrote:

 On 2013-07-10 20:22, Ali Çehreli wrote:

 And to be pedantic, length comes first:

 struct Array (T)
 {
  size_t length;
  T* ptr;
 }

 I thought ptr came first, that's the reason you could cast
to the
 pointer type. Not that one should do that. Perhaps there's
some
 compiler/runtime magic involved.

There must be little magic and that magic should be the same as 
getting the .ptr property. Otherwise, the value of a struct 
object cannot be casted to pointer type:


struct S
{
int *p;
}

auto s = S();
int *p = cast(int*)s;

Error: e2ir: cannot cast s of type S to type int*

Ali


In context of slices cast(int*)arr is essentially s.ptr. There is 
no magic but accessing right field of struct.


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-11 Thread Ali Çehreli

On 07/11/2013 10:20 AM, Maxim Fomin wrote:

 On Thursday, 11 July 2013 at 17:07:18 UTC, Ali Çehreli wrote:

 that magic should be the same as
 getting the .ptr property.

Yes.

 In context of slices cast(int*)arr is essentially s.ptr.

And yes. :)

Ali



pointers, assignments, Garbage Collection Oh My?

2013-07-10 Thread JohnnyK
I hope you like the subject matter and I hope it is not too 
simplistic or have been answered before.
  Anyway I have a question about how the garbage collector works 
in a very specific situation.  When passing string type to a 
function in a shared library or DLL and assigning it to a 
variable of type string inside the function and returning the 
internal string.  Such as this.


export string mytest(string tstStr)
{
  string st = tstStr;
  /* abbreviated to protect the innocent but other operations
 such as concatenating and deleting may be done to st before 
the return

  */
  return st;
}

Is the string type a pointer or is it something else?  In the 
line where tstStr is assigned to st does it copy the address in 
tstStr to st or does it copy the value in tstStr?  I am just a 
bit confused about string types since I come from a C background 
and C has no type like this.  Also what is returned by this 
function?  Does this function return a pointer or the contents of 
an array?  If I do export this what does it do to the Garbage 
Collection?  Does the Garbage Collection collect tstStr or st?  
Also notice the comment in the function.




Re: pointers, assignments, Garbage Collection Oh My?

2013-07-10 Thread Jacob Carlborg

On 2013-07-10 19:18, JohnnyK wrote:

I hope you like the subject matter and I hope it is not too simplistic
or have been answered before.
   Anyway I have a question about how the garbage collector works in a
very specific situation.  When passing string type to a function in a
shared library or DLL and assigning it to a variable of type string
inside the function and returning the internal string.  Such as this.

export string mytest(string tstStr)
{
   string st = tstStr;
   /* abbreviated to protect the innocent but other operations
  such as concatenating and deleting may be done to st before the
return
   */
   return st;
}

Is the string type a pointer or is it something else?  In the line where
tstStr is assigned to st does it copy the address in tstStr to st or
does it copy the value in tstStr?  I am just a bit confused about string
types since I come from a C background and C has no type like this.
Also what is returned by this function?  Does this function return a
pointer or the contents of an array?  If I do export this what does it
do to the Garbage Collection?  Does the Garbage Collection collect
tstStr or st? Also notice the comment in the function.


A string in D, and all arrays, is a struct looking like this:

struct Array (T)
{
T* ptr;
size_t length;
}

string happens to be an alias looking like this:

alias immutable(char)[] string;

That means you can do all operations on a string except assigning to 
specific elements:


string str = foo;
str ~= bar; // ok
auto str2 = str ~ foo; // ok
str[0] == f; // ok
str[0] = 'a'; // error, cannot assign to immutable
auto str3 = str[1 .. 3]; // ok

It won't collect anything along as it is scope. If a variable goes out 
of scope and nothing else points to that data it will collect it 
(eventually).


Hope this helps a bit.

--
/Jacob Carlborg


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-10 Thread Namespace

A string in D, and all arrays, is a struct looking like this:

struct Array (T)
{
T* ptr;
size_t length;
}


I always thought it looks like this:

struct Array(T) {
T* ptr;
size_t length, capacity;
}


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-10 Thread Sean Kelly
On Jul 10, 2013, at 10:45 AM, Namespace rswhi...@googlemail.com wrote:

 A string in D, and all arrays, is a struct looking like this:
 
 struct Array (T)
 {
T* ptr;
size_t length;
 }
 
 I always thought it looks like this:
 
 struct Array(T) {
T* ptr;
size_t length, capacity;
 }

Sadly, no.  The only way to determine the capacity of an array is to query the 
GC.

Re: pointers, assignments, Garbage Collection Oh My?

2013-07-10 Thread Ali Çehreli

On 07/10/2013 11:10 AM, Sean Kelly wrote:

 On Jul 10, 2013, at 10:45 AM, Namespace rswhi...@googlemail.com wrote:

 A string in D, and all arrays, is a struct looking like this:

 struct Array (T)
 {
 T* ptr;
 size_t length;
 }

 I always thought it looks like this:

 struct Array(T) {
 T* ptr;
 size_t length, capacity;
 }

 Sadly, no.  The only way to determine the capacity of an array is to 
query the GC.



And to be pedantic, length comes first:

struct Array (T)
{
size_t length;
T* ptr;
}

Which is actually property-like because assigning to length does pretty 
complex stuff. So the member cannot be named as 'length':


struct Array (T)
{
size_t length_;
T* ptr;
}

Anyway... :)

Ali



Re: pointers, assignments, Garbage Collection Oh My?

2013-07-10 Thread JohnnyK

On Wednesday, 10 July 2013 at 18:22:24 UTC, Ali Çehreli wrote:

On 07/10/2013 11:10 AM, Sean Kelly wrote:

 On Jul 10, 2013, at 10:45 AM, Namespace
rswhi...@googlemail.com wrote:

 A string in D, and all arrays, is a struct looking like
this:

 struct Array (T)
 {
 T* ptr;
 size_t length;
 }

 I always thought it looks like this:

 struct Array(T) {
 T* ptr;
 size_t length, capacity;
 }

 Sadly, no.  The only way to determine the capacity of an
array is to query the GC.


And to be pedantic, length comes first:

struct Array (T)
{
size_t length;
T* ptr;
}

Which is actually property-like because assigning to length 
does pretty complex stuff. So the member cannot be named as 
'length':


struct Array (T)
{
size_t length_;
T* ptr;
}

Anyway... :)

Ali


Reminds me of how Delphi (aka Pascal) strings are work.  Thanks 
everyone this answers some of my questions.  Now what about when 
the return type of a function is a string?  Is D returning the 
pointer to the string structure or is it returning the structure?


Re: pointers, assignments, Garbage Collection Oh My?

2013-07-10 Thread H. S. Teoh
On Wed, Jul 10, 2013 at 07:45:25PM +0200, Namespace wrote:
 A string in D, and all arrays, is a struct looking like this:
 
 struct Array (T)
 {
 T* ptr;
 size_t length;
 }
 
 I always thought it looks like this:
 
 struct Array(T) {
 T* ptr;
 size_t length, capacity;
 }

Nope, the capacity is an attribute of the GC memory that the array is
pointing to, not the array itself, which is merely a slice of this GC
memory.

When you append to an array, basically what happens is the GC is asked
is this the end of the GC block and can we extend it please? If it's
not the end of the GC block, a new block is allocated; otherwise, it is
extended, then the new data is written into it.


T

-- 
Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at 
it. -- Pete Bleackley


  1   2   >