Re: Garbage Collector profiling and the dynamic array reserve() function

[bauss via Digitalmars-d-learn]

On Wednesday, 18 October 2017 at 15:39:43 UTC, Steven 
Schveighoffer wrote:

On 10/18/17 1:40 AM, Tony wrote:
On Tuesday, 17 October 2017 at 13:27:24 UTC, Steven 
Schveighoffer wrote:




I don't know what "allocations" represents, but reserve 
actually calls gc_malloc, and the others do not (the space is 
available to expand into the block). There should be only one 
allocation IMO.




So there should be a bug report written for this?



It all depends on what "allocations" means. I'd wait to find 
out from someone who is familiar with the GC profiling.


-Steve


I don't have a lot of clues on how the GC profiling work, but 
looking at reserve() it calls mem.xmalloc() for allocations which 
in fact calls GC.malloc().


Looking at the profiling for GC though:
https://github.com/dlang/dmd/blob/69567a32c5bffae5513b41e7691c91b50766b552/src/ddmd/e2ir.d#L5952

It doesn't look like there's anything for array reserve calls, 
unless:

 [ RTLSYM_ALLOCMEMORY, RTLSYM_TRACEALLOCMEMORY ]
are triggered from the allocations done in reserve(), but I have 
no idea about that.

Re: Garbage Collector profiling and the dynamic array reserve() function

[Steven Schveighoffer via Digitalmars-d-learn]

On 10/18/17 1:40 AM, Tony wrote:

On Tuesday, 17 October 2017 at 13:27:24 UTC, Steven Schveighoffer wrote:



I don't know what "allocations" represents, but reserve actually calls 
gc_malloc, and the others do not (the space is available to expand 
into the block). There should be only one allocation IMO.




So there should be a bug report written for this?



It all depends on what "allocations" means. I'd wait to find out from 
someone who is familiar with the GC profiling.


-Steve

Re: Garbage Collector profiling and the dynamic array reserve() function

[Tony via Digitalmars-d-learn]

On Tuesday, 17 October 2017 at 13:27:24 UTC, Steven Schveighoffer 
wrote:




I don't know what "allocations" represents, but reserve 
actually calls gc_malloc, and the others do not (the space is 
available to expand into the block). There should be only one 
allocation IMO.


-Steve


So there should be a bug report written for this?

Re: Garbage Collector profiling and the dynamic array reserve() function

[Steven Schveighoffer via Digitalmars-d-learn]

On 10/17/17 2:14 AM, Tony wrote:

Found this unanswered question on StackOverflow.

This program:

import std.stdio;

void add(ref int[] data)
{
     data ~= 1;
     data ~= 2;
}

void main()
{
     int[] a;
     writeln("capacity:",a.capacity);
     auto cap = a.reserve(1000); // allocated may be more than requested
     assert(cap >= 1000);
     assert(cap == a.capacity);
     writeln("capacity:",a.capacity);
     a.add();
     writeln(a);

}

compiled with "dmd -profile=gc"

has this output in profilegc.log

bytes allocated, allocations, type, function, file:line
   4  1    int[] profiling.add profiling.d:8
   4  1    int[] profiling.add profiling.d:7

The question is: why doesn't using reserve() cause an allocation to be 
shown?


I don't know what "allocations" represents, but reserve actually calls 
gc_malloc, and the others do not (the space is available to expand into 
the block). There should be only one allocation IMO.


-Steve

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Saturday, 29 April 2017 at 10:54:02 UTC, bachmeier wrote:
Many invested in Rust and C++ will look for arguments to 
support staying with their language. I've come to the 
conclusion that the D community is mostly to blame for not 
making a good case to the other group that are open to D, but 
for technical reasons or simply personal preference don't like 
GC, that D is still an option. There's no excuse for not making 
it easy to evaluate one's options for GC-less programming if we 
support that style of programming.


Of course. It is useful for D-users who are looking for 
alternatives to the GC if written for that group. Finding 
complete and up to date information is always the biggest 
challenge for newbies.


Just don't expect it to have an effect on reddit users...

Re: Garbage Collector?

[bachmeier via Digitalmars-d]

On Friday, 28 April 2017 at 19:49:35 UTC, Ola Fosheim Grøstad 
wrote:
On Friday, 28 April 2017 at 17:48:47 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 17:42:18 UTC, bachmeier wrote:
I'm hoping to put all information in one place. Then when 
someone on Reddit or HN or here starts making claims about 
the GC, I can give them one link that shows all of their 
options.


That's nice. Just get your hopes up for it having an effect.


Typo, I meant "don't"... Sloppy of me. Documentation is nice, 
but:


1. People will complain that it isn't possible.

2. When possible people will complain that it isn't in the 
standard library.


3. When in "std" people will complain that not enough libraries 
use it.


4. When libraries use it people will complain that it doesn't 
work with older libs.


5. When older libs have been rewritten to support it they will 
complain that it is better in Rust and C++ and not compatible 
with Rust and C++.


Anyway, my main point is that programmers coming from such 
languages will most certainly complain if it isn't in the 
standard library because of interoperability between libraries, 
but that is basically just the bottom of the hill that you have 
to climb to get to a level where people stop complaining.


Many invested in Rust and C++ will look for arguments to support 
staying with their language. I've come to the conclusion that the 
D community is mostly to blame for not making a good case to the 
other group that are open to D, but for technical reasons or 
simply personal preference don't like GC, that D is still an 
option. There's no excuse for not making it easy to evaluate 
one's options for GC-less programming if we support that style of 
programming.

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Saturday, 29 April 2017 at 09:24:35 UTC, Ola Fosheim Grøstad 
wrote:
On Saturday, 29 April 2017 at 08:45:26 UTC, Moritz Maxeiner 
wrote:

On Saturday, 29 April 2017 at 07:26:45 UTC, Timon Gehr wrote:
I don't doubt that, but the implicit generalization is 
"multiple pointer types are necessarily always a royal PITA".


The "implicit generalization" is your interpretation, though.


No, it was brought up in a thread as an argument against having 
multiple pointer types.


It was brought up in that thread as an example of multiple 
pointer types having (I assume unintended) negative consequences. 
That's not the same as implying that *all* occurrences of 
multiple pointer types *will* have such negative consequences; at 
most, it implies that you have to be very careful when designing 
them, so as to avoid such consequences (and this latter part is 
*my* interpretation).


The thread was not about near/far pointers or segmented memory 
models.


I am aware; it was originally about the viability of automatic 
reference counting in D, and its potential benefits/drawbacks 
compared to garbage collection.

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Saturday, 29 April 2017 at 08:45:26 UTC, Moritz Maxeiner wrote:

On Saturday, 29 April 2017 at 07:26:45 UTC, Timon Gehr wrote:
I don't doubt that, but the implicit generalization is 
"multiple pointer types are necessarily always a royal PITA".


The "implicit generalization" is your interpretation, though.


No, it was brought up in a thread as an argument against having 
multiple pointer types. The thread was not about near/far 
pointers or segmented memory models.

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Saturday, 29 April 2017 at 07:26:45 UTC, Timon Gehr wrote:

On 28.04.2017 23:52, H. S. Teoh via Digitalmars-d wrote:
On Fri, Apr 28, 2017 at 09:50:49PM +, Atila Neves via 
Digitalmars-d wrote:
On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad 
wrote:
On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim 
Grøstad wrote:
«Back in the old DOS days, there were multiple pointer types 
(near
and far). Programmers put up with that because it was the 
only way,

but they HATED HATED HATED it.»


It's true, we did. It was awful.

[...]

I remember working with that.  It was a royal PITA.


T



I don't doubt that, but the implicit generalization is 
"multiple pointer types are necessarily always a royal PITA".


The "implicit generalization" is your interpretation, though.

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Saturday, 29 April 2017 at 07:15:36 UTC, Timon Gehr wrote:

On 28.04.2017 17:43, Moritz Maxeiner wrote:

[...]

No. Every single thread I read in the last couple of years 
ended with
Walter pointing out issues that need to be "hashed out" and 
then nobody

doing it.


This is not the full story though. Among Walter's constraints 
for an acceptable solution always were:


1. cannot add new syntax


I've dug through what records I know of surrounding these 
discussions [1][2][3][4][5] and could not come up with a 
statement of his that supports this hard-line claim. You may be 
right, considering that adding new syntax is a non-trivial 
change, but I'd need the citation.




2. cannot add polymorphism to the type system (because it is 
"hard to understand")


Same as above.

[1] http://lists.puremagic.com/pipermail/dlang-study/
[2] 
https://forum.dlang.org/thread/kpgilxyyrrluxpepe...@forum.dlang.org
[3] 
https://forum.dlang.org/post/kluaojijixhwigouj...@forum.dlang.org

[4] http://forum.dlang.org/thread/oboaa2$17oa$1...@digitalmars.com
[5] http://forum.dlang.org/thread/nu00a6$t5i$1...@digitalmars.com

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Saturday, 29 April 2017 at 07:26:45 UTC, Timon Gehr wrote:
I don't doubt that, but the implicit generalization is 
"multiple pointer types are necessarily always a royal PITA". 
Not true. scope has worse usability than a scoped pointer type.


Yes. In this context it is was more about the GC lifetime 
management which requires:


1. The ability to tell the GC about explicit parent-child 
relationships so that the GC destruction order becomes 
deterministic.


2. Precise collection.

Re: Garbage Collector?

[Timon Gehr via Digitalmars-d]

On 28.04.2017 23:52, H. S. Teoh via Digitalmars-d wrote:

On Fri, Apr 28, 2017 at 09:50:49PM +, Atila Neves via Digitalmars-d wrote:

On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad wrote:

On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad wrote:
«Back in the old DOS days, there were multiple pointer types (near
and far). Programmers put up with that because it was the only way,
but they HATED HATED HATED it.»


It's true, we did. It was awful.

[...]

I remember working with that.  It was a royal PITA.


T



I don't doubt that, but the implicit generalization is "multiple pointer 
types are necessarily always a royal PITA". Not true. scope has worse 
usability than a scoped pointer type.

Re: Garbage Collector?

[Timon Gehr via Digitalmars-d]

On 28.04.2017 17:43, Moritz Maxeiner wrote:

On Friday, 28 April 2017 at 14:59:46 UTC, Ola Fosheim Grøstad wrote:

On Friday, 28 April 2017 at 09:40:07 UTC, Moritz Maxeiner wrote:

I'm sorry, but that's just plain wrong. D does not have ownership
pointers because nobody that wants them has stepped up and
1) Done the work of drafting an informal proposal that *actually
deals with _all_ of the issues involved*


Wrong. This has been discussed and hashed out to death over and over
again. The solutions are on the table.


No. Every single thread I read in the last couple of years ended with
Walter pointing out issues that need to be "hashed out" and then nobody
doing it.


This is not the full story though. Among Walter's constraints for an 
acceptable solution always were:


1. cannot add new syntax

2. cannot add polymorphism to the type system (because it is "hard to 
understand")


1. is inconvenient and I think what we get with 'scope' is approximately 
the best one can do given 2.

Re: Garbage Collector?

[Ola Fosheim Grostad via Digitalmars-d]

On Friday, 28 April 2017 at 21:21:13 UTC, jmh530 wrote:
To be fair, C++ effectively has multiple pointer types too with 
raw pointers, unique_ptr, shared_ptr, and weak_ptr. However, 
each of the extra ones has a unique purpose and are opt-in. As 
a result, people happily use them when it makes their lives 
easier.


Yes, they are not language types though, so no special effect on 
the compiler or runtime. The language types are pointers, 
 and &


By contrast, C++/CLI (I'm more familiar with that than managed 
C++) has pointer to managed heap and pointer to unmanaged heap. 
The concepts overlap more.


Yes, and I assume those are language types so that the compiler 
and runtime can take advantage of it?

Re: Garbage Collector?

[H. S. Teoh via Digitalmars-d]

On Fri, Apr 28, 2017 at 09:50:49PM +, Atila Neves via Digitalmars-d wrote:
> On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad wrote:
> > On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad wrote:
> > «Back in the old DOS days, there were multiple pointer types (near
> > and far). Programmers put up with that because it was the only way,
> > but they HATED HATED HATED it.»
> 
> It's true, we did. It was awful.
[...]

I remember working with that.  It was a royal PITA.


T

-- 
Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at 
it. -- Pete Bleackley

Re: Garbage Collector?

[Atila Neves via Digitalmars-d]

On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad 
wrote:
On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad 
wrote:
«Back in the old DOS days, there were multiple pointer types 
(near and far). Programmers put up with that because it was the 
only way, but they HATED HATED HATED it.»


It's true, we did. It was awful.

Atila

Re: Garbage Collector?

[jmh530 via Digitalmars-d]

On Friday, 28 April 2017 at 20:21:34 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 20:13:58 UTC, Moritz Maxeiner wrote:
Both of these, however, show only that he doesn't seem to 
personally like multiple pointer types (and consequently 
doesn't seem to have any interest in working on them himself); 
that's not the same as him claiming that it "is a disaster" 
(in general, which is what you were implying).


There are other threads. This is a recurring topic...



To be fair, C++ effectively has multiple pointer types too with 
raw pointers, unique_ptr, shared_ptr, and weak_ptr. However, each 
of the extra ones has a unique purpose and are opt-in. As a 
result, people happily use them when it makes their lives easier.


By contrast, C++/CLI (I'm more familiar with that than managed 
C++) has pointer to managed heap and pointer to unmanaged heap. 
The concepts overlap more.

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Friday, 28 April 2017 at 20:21:34 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 20:13:58 UTC, Moritz Maxeiner wrote:
Both of these, however, show only that he doesn't seem to 
personally like multiple pointer types (and consequently 
doesn't seem to have any interest in working on them himself); 
that's not the same as him claiming that it "is a disaster" 
(in general, which is what you were implying).


There are other threads. This is a recurring topic...


I am aware...



What you consider not pointless is your business, again, but 
if you don't try to get it in the core language, you have no 
foundation to complain that's it's not in there.


There are no example I know of where something has gone into 
the D language that is going against the aesthetics Walter 
value... Which isn't surprising.


There are also few examples of new features going in at the 
language level based on external DIPs. Do you have any good 
examples? (Not standard lib, but language features).


I myself know of no examples where people have actually done the 
work up to and including the point where there were no technical 
issues left and were then rejected.

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Friday, 28 April 2017 at 20:13:58 UTC, Moritz Maxeiner wrote:
Both of these, however, show only that he doesn't seem to 
personally like multiple pointer types (and consequently 
doesn't seem to have any interest in working on them himself); 
that's not the same as him claiming that it "is a disaster" (in 
general, which is what you were implying).


There are other threads. This is a recurring topic...

What you consider not pointless is your business, again, but if 
you don't try to get it in the core language, you have no 
foundation to complain that's it's not in there.


There are no example I know of where something has gone into the 
D language that is going against the aesthetics Walter value... 
Which isn't surprising.


There are also few examples of new features going in at the 
language level based on external DIPs. Do you have any good 
examples? (Not standard lib, but language features).

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Friday, 28 April 2017 at 19:41:15 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 15:43:22 UTC, Moritz Maxeiner wrote:
On Friday, 28 April 2017 at 14:59:46 UTC, Ola Fosheim Grøstad 
wrote:
Walter's position has always been that having more than a 
single pointer type is a disaster.


None of the threads I've read in the last couple of years 
regarding that support that claim.


He has restated this position many many times... Random 
snippets:


«Microsoft's Managed C++ had two pointer types, and it went 
over like a lead zeppelin.

»

http://forum.dlang.org/post/mclqt1$1e5n$1...@digitalmars.com


«Back in the old DOS days, there were multiple pointer types 
(near and far). Programmers put up with that because it was the 
only way, but they HATED HATED HATED it.»


http://forum.dlang.org/post/mcnv9u$e8p$1...@digitalmars.com



I had not read these, thank you :)
Both of these, however, show only that he doesn't seem to 
personally like multiple pointer types (and consequently doesn't 
seem to have any interest in working on them himself); that's not 
the same as him claiming that it "is a disaster" (in general, 
which is what you were implying).




You can easily find more... No point in trying to get that into 
the core language (but it is necessary to get proper 
destruction of GC managed objects in a reasonable way).


What you consider not pointless is your business, again, but if 
you don't try to get it in the core language, you have no 
foundation to complain that's it's not in there.

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Friday, 28 April 2017 at 17:48:47 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 17:42:18 UTC, bachmeier wrote:
I'm hoping to put all information in one place. Then when 
someone on Reddit or HN or here starts making claims about the 
GC, I can give them one link that shows all of their options.


That's nice. Just get your hopes up for it having an effect.


Typo, I meant "don't"... Sloppy of me. Documentation is nice, but:

1. People will complain that it isn't possible.

2. When possible people will complain that it isn't in the 
standard library.


3. When in "std" people will complain that not enough libraries 
use it.


4. When libraries use it people will complain that it doesn't 
work with older libs.


5. When older libs have been rewritten to support it they will 
complain that it is better in Rust and C++ and not compatible 
with Rust and C++.


Anyway, my main point is that programmers coming from such 
languages will most certainly complain if it isn't in the 
standard library because of interoperability between libraries, 
but that is basically just the bottom of the hill that you have 
to climb to get to a level where people stop complaining.

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Friday, 28 April 2017 at 15:43:22 UTC, Moritz Maxeiner wrote:
On Friday, 28 April 2017 at 14:59:46 UTC, Ola Fosheim Grøstad 
wrote:
Walter's position has always been that having more than a 
single pointer type is a disaster.


None of the threads I've read in the last couple of years 
regarding that support that claim.


He has restated this position many many times... Random snippets:

«Microsoft's Managed C++ had two pointer types, and it went over 
like a lead zeppelin.

»

http://forum.dlang.org/post/mclqt1$1e5n$1...@digitalmars.com


«Back in the old DOS days, there were multiple pointer types 
(near and far). Programmers put up with that because it was the 
only way, but they HATED HATED HATED it.»


http://forum.dlang.org/post/mcnv9u$e8p$1...@digitalmars.com


You can easily find more... No point in trying to get that into 
the core language (but it is necessary to get proper destruction 
of GC managed objects in a reasonable way).

Re: Garbage Collector?

[H. S. Teoh via Digitalmars-d]

On Fri, Apr 28, 2017 at 04:03:18PM +, bachmeier via Digitalmars-d wrote:
> On Friday, 28 April 2017 at 15:23:18 UTC, H. S. Teoh wrote:
> 
> > you could save yourself the bug by writing:
> > 
> > auto x = malloc(...);
> > scope(exit) free(x);
> > // ... however many pages of stuff you want, you don't have to
> > // remember to write free() afterwards!
> > 
> > Yes, D comes with a GC... doesn't mean you have to use it if you
> > don't want to, though!
> 
> I usually use the GC, so I have limited knowledge in this area. How
> common is this pattern in D code? Is it better than using reference
> counted structs? Is there any advantage to using the GC in this
> scenario?

To be honest, in my own D code I'm not overly concerned with the GC, and
even when I am, and want to control when / how often the GC collects, I
usually just use GC.disable() and then GC.collect() explicitly.  I've
seen performance gains of about 30-40% just by carefully reducing the
frequency of GC collections (IMO the default is a bit too eager), but of
course that depends on exactly what your application is doing.  In my
case, that was in the context of batch-oriented, CPU-intensive
computations that allocate often and mostly keeps live data.  YMMV if
your program is doing something else or has other patterns of memory
access.

But my point was that, if you really wanted to, you *could* use C-style
memory management in D code. D won't stop you from doing that. The
malloc heap is distinct from the GC heap so you won't run into
conflicts.  Of course, that also means you can't use D features that
currently require the GC, such as closures and dynamic arrays. But if
you're doing C-style memory management you probably already want to
implement your own way of handling array allocations and closure
functionality anyway. You can still pass slices of things around to
things like Phobos range algorithms (excepting the few that might
allocate -- hence marking your code with @nogc for the compiler to
enforce this), so generally a decent chunk of Phobos should still be
usable.

Even Phobos itself uses malloc/free directly in several places, for
various reasons, so I wouldn't say it's exactly a foreign thing to do in
D code.


> I would like to add this info to the wiki (I don't seen it there).

That would be very nice, thanks!


T

-- 
Give a man a fish, and he eats once. Teach a man to fish, and he will sit 
forever.

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Friday, 28 April 2017 at 17:42:18 UTC, bachmeier wrote:
I'm hoping to put all information in one place. Then when 
someone on Reddit or HN or here starts making claims about the 
GC, I can give them one link that shows all of their options.


That's nice. Just get your hopes up for it having an effect. One 
of the key points of having it in the core distribution 
(compiler) is for library interoperability. Using multiple 
solutions ends up in chaos in real world projects...

Re: Garbage Collector?

[bachmeier via Digitalmars-d]

On Friday, 28 April 2017 at 17:06:51 UTC, jmh530 wrote:

On Friday, 28 April 2017 at 16:03:18 UTC, bachmeier wrote:


I usually use the GC, so I have limited knowledge in this 
area. How common is this pattern in D code? Is it better than 
using reference counted structs? Is there any advantage to 
using the GC in this scenario?


I would like to add this info to the wiki (I don't seen it 
there).


You can also use automem
https://dlang.org/blog/2017/04/28/automem-hands-free-raii-for-d/


I'm hoping to put all information in one place. Then when someone 
on Reddit or HN or here starts making claims about the GC, I can 
give them one link that shows all of their options. There's this 
page:

https://wiki.dlang.org/Memory_Management
but that isn't comprehensive or as to-the-point as it needs to be.

Re: Garbage Collector?

[jmh530 via Digitalmars-d]

On Friday, 28 April 2017 at 16:03:18 UTC, bachmeier wrote:


I usually use the GC, so I have limited knowledge in this area. 
How common is this pattern in D code? Is it better than using 
reference counted structs? Is there any advantage to using the 
GC in this scenario?


I would like to add this info to the wiki (I don't seen it 
there).


You can also use automem
https://dlang.org/blog/2017/04/28/automem-hands-free-raii-for-d/

Re: Garbage Collector?

[bachmeier via Digitalmars-d]

On Friday, 28 April 2017 at 15:23:18 UTC, H. S. Teoh wrote:


you could save yourself the bug by writing:

auto x = malloc(...);
scope(exit) free(x);
// ... however many pages of stuff you want, you don't have to
// remember to write free() afterwards!

Yes, D comes with a GC... doesn't mean you have to use it if 
you don't want to, though!


I usually use the GC, so I have limited knowledge in this area. 
How common is this pattern in D code? Is it better than using 
reference counted structs? Is there any advantage to using the GC 
in this scenario?


I would like to add this info to the wiki (I don't seen it there).

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Friday, 28 April 2017 at 14:59:46 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 09:40:07 UTC, Moritz Maxeiner wrote:
I'm sorry, but that's just plain wrong. D does not have 
ownership pointers because nobody that wants them has stepped 
up and
1) Done the work of drafting an informal proposal that 
*actually deals with _all_ of the issues involved*


Wrong. This has been discussed and hashed out to death over and 
over again. The solutions are on the table.


No. Every single thread I read in the last couple of years ended 
with Walter pointing out issues that need to be "hashed out" and 
then nobody doing it.




Walter's position has always been that having more than a 
single pointer type is a disaster.


None of the threads I've read in the last couple of years 
regarding that support that claim.




That makes further work on it pointless.


What you consider worth working on is your business, of course :)

Re: Garbage Collector?

[H. S. Teoh via Digitalmars-d]

On Fri, Apr 28, 2017 at 09:01:03AM +, Moritz Maxeiner via Digitalmars-d 
wrote:
> On Friday, 28 April 2017 at 07:35:00 UTC, Ben wrote:
[...]
> > Is it so hard for developers when you declare a variable, to later
> > also clean it up???
> > 
> > var x = 1;
> > // Do work
> > x.free;
> 
> If you write it like that, yes, because often it's not just one such
> make/dispose pair per scope, but multiple, possibly overlapping ones
> and people make mistakes. And the more complex a piece of code gets
> the harder it becomes to decipher such pairs and/or decide if the
> "closing" dispose is missing.
> This is one of the reasons why scope guards are good:
> 
> var x = 1;
> scope (exit) x.free
> // Do work
> 
> This, as code becomes more complex, allows for much easier reading
> (and understanding) of lifetimes.
[...]

Elephant in the room: D lets you call the C library's malloc() and
free(). If you absolute insist that you don't want to use the GC, go
right ahead and import core.c.stdlib, and malloc and free away.  As
mentioned above, D's scope guards will even help you avoid mistakes by
keeping allocation and free in the same scope together, i.e., instead of
writing:

auto x = malloc(...);
// do stuff
// and more stuff
// and more stuff
// so many pages of stuff you forgot about x
free(x);// very likely you'll forget this by now

you could save yourself the bug by writing:

auto x = malloc(...);
scope(exit) free(x);
// ... however many pages of stuff you want, you don't have to
// remember to write free() afterwards!

Yes, D comes with a GC... doesn't mean you have to use it if you don't
want to, though!


T

-- 
Doubt is a self-fulfilling prophecy.

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Friday, 28 April 2017 at 09:40:07 UTC, Moritz Maxeiner wrote:
I'm sorry, but that's just plain wrong. D does not have 
ownership pointers because nobody that wants them has stepped 
up and
1) Done the work of drafting an informal proposal that 
*actually deals with _all_ of the issues involved*


Wrong. This has been discussed and hashed out to death over and 
over again. The solutions are on the table.


Walter's position has always been that having more than a single 
pointer type is a disaster.


That makes further work on it pointless.

(Although, arguably, D kinda has several pointer types already).

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Friday, 28 April 2017 at 09:12:03 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 09:02:19 UTC, Moritz Maxeiner wrote:
On Friday, 28 April 2017 at 08:26:28 UTC, Ola Fosheim Grøstad 
wrote:

If it isn't sound then it isn't worth mentioning...


So you claim...


Actually, unsound lifetime management is worse than nothing as 
far as correctness goes... so I guess it is worth mentioning as 
a warning rather than a recommendation.


Actually, applying "sound" or "unsound" to D's GC lifetime 
management in general is worse than realizing that whether using 
it for a specific use case is sound or unsound is entirely 
dependent on that use case... so I guess it is worth mentioning 
as a tool a developer should read up on.
Also, you seem to imply that I made a recommendation to use the 
GC's lifetime management. I did not.





If D had ownership pointers then it would've been possible to 
sort this out, but D is perpetually locked to the current 
pointer/GC model for reasons that aren't particularly 
convincing, i.e. beliefs.


I'm sorry, but that's just plain wrong. D does not have ownership 
pointers because nobody that wants them has stepped up and
1) Done the work of drafting an informal proposal that *actually 
deals with _all_ of the issues involved*

2) Developed that informal proposal into a DIP
3) Wrote the implementation and sent a PR
Every time it comes up there are ideas, or sometimes even 
reasonably well formed informal proposals, but none of the 
proponents ever actually follows up and improves such proposals 
to the point where all the issues are dealt with. Claiming that D 
doesn't have ownership mechanics because of beliefs when none of 
the (vocal) proponents are willing to actually get down to it and 
*do the work* is, frankly, asinine; do you generally expect 
people to do things for you they aren't interested in without 
recompense?

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Friday, 28 April 2017 at 09:02:19 UTC, Moritz Maxeiner wrote:
On Friday, 28 April 2017 at 08:26:28 UTC, Ola Fosheim Grøstad 
wrote:

If it isn't sound then it isn't worth mentioning...


So you claim...


Actually, unsound lifetime management is worse than nothing as 
far as correctness goes... so I guess it is worth mentioning as a 
warning rather than a recommendation.


If D had ownership pointers then it would've been possible to 
sort this out, but D is perpetually locked to the current 
pointer/GC model for reasons that aren't particularly convincing, 
i.e. beliefs.

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Friday, 28 April 2017 at 08:26:28 UTC, Ola Fosheim Grøstad 
wrote:

On Friday, 28 April 2017 at 08:25:01 UTC, Moritz Maxeiner wrote:
On Friday, 28 April 2017 at 04:24:43 UTC, Ola Fosheim Grostad 
wrote:
On Thursday, 27 April 2017 at 22:43:56 UTC, Moritz Maxeiner 
wrote:

Working on the memory chunk layer is memory management.
Working on the object layer is object lifetime management.
D offers you both automatic memory management and automatic 
lifetime management via its GC.


D offers sound automatic lifetime management? Since when?


Why are you implicitly adding adjectives to my sentence that 
change its meaning?


If it isn't sound then it isn't worth mentioning...


So you claim...

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Friday, 28 April 2017 at 07:35:00 UTC, Ben wrote:
On Thursday, 27 April 2017 at 22:43:56 UTC, Moritz Maxeiner 
wrote:
D just does this transparently for you by default. If you 
already know the exact or maximum size, you can allocate 
*once* (not 6 times) using `new` and `.reserve` respectively 
*before* entering the loop, like that article explains in 
depth.


You seem to be missing the fact that i pointed this out. The 
fact that the GC might have done up to 6 collection rounds in 
that loop is "ludicrous".


You wrote:
I if did my own memory management, the variable cleanup will 
have been done in one go, right after the loop. Simply more 
efficient.


You wrote about cleaning up *after* the loop, *not* about 
allocating *before* the loop. They are semantically related (as 
allocating with the GC *might* kick off a collection cycle), but 
not the same thing.




In my experience most people's aversion to GCs can be aptly 
described by the German proverb "Was der Bauer nicht kennt, 
das frisst er nicht" (the meaning of which being that most 
people generally like living in comfort zones and rarely - if 
ever - dare to leave them of their own free will; that 
includes developers, sadly).


Unfortunately i do have plenty of experience with GC kicking in 
on the wrong moments ( or not at the right moments, people 
forget that one ).


In which language(s)? Because in D it's both easy and simple to 
avoid that (Use @nogc or compile with `-vgc` and inspect).




Maybe its my opinion only but a good language will not put 
anything in the way of the developer but will point out 
mistakes.


Safety properties may be what you're looking for. See @safe, 
that's an area of D a lot of work is currently being done on.




The Rust compiler is not a bad example but it can be taken a 
step more. Is it so hard for developers when you declare a 
variable, to later also clean it up???


var x = 1;
// Do work
x.free;


If you write it like that, yes, because often it's not just one 
such make/dispose pair per scope, but multiple, possibly 
overlapping ones and people make mistakes. And the more complex a 
piece of code gets the harder it becomes to decipher such pairs 
and/or decide if the "closing" dispose is missing.

This is one of the reasons why scope guards are good:

var x = 1;
scope (exit) x.free
// Do work

This, as code becomes more complex, allows for much easier 
reading (and understanding) of lifetimes.





Easy ... Sure, it becomes a little bit more tricky with 
ownership but that is where the compiler can help and simply 
state: "Hey, you forgot this variable, it does not seem to be 
used beyond this point".


Since the GC is memory safe by definition (sans bugs) you'd only 
need this for where you aren't using it. And for the rest I'm not 
sure it can be done sanely: How's the compiler to know, e.g., 
which functions from a C API (or C++) you call into allocate or 
deallocate memory? Introduce yet another keyword or @property to 
signal cross-language memory management? If you ignore 
cross-language calls and only remain within D, I suppose it might 
be viable to write a semantic pass for it, but from my 
perspective, the cross-language calls are the *much* bigger 
threat to memory safety.


Just calling the x.free seem to be too much work for developers 
these days.


In my experience reading about such occurences, this usually 
happens when the lifetime of the object the memory is used for 
has become unclear.




Up to now i found very few languages that did this correctly.


I don't share your assessment of what's correct, but yes, we 
digress.

Re: Garbage Collector?

[Ola Fosheim Grøstad via Digitalmars-d]

On Friday, 28 April 2017 at 08:25:01 UTC, Moritz Maxeiner wrote:
On Friday, 28 April 2017 at 04:24:43 UTC, Ola Fosheim Grostad 
wrote:
On Thursday, 27 April 2017 at 22:43:56 UTC, Moritz Maxeiner 
wrote:

Working on the memory chunk layer is memory management.
Working on the object layer is object lifetime management.
D offers you both automatic memory management and automatic 
lifetime management via its GC.


D offers sound automatic lifetime management? Since when?


Why are you implicitly adding adjectives to my sentence that 
change its meaning?


If it isn't sound then it isn't worth mentioning...

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Friday, 28 April 2017 at 04:24:43 UTC, Ola Fosheim Grostad 
wrote:
On Thursday, 27 April 2017 at 22:43:56 UTC, Moritz Maxeiner 
wrote:

Working on the memory chunk layer is memory management.
Working on the object layer is object lifetime management.
D offers you both automatic memory management and automatic 
lifetime management via its GC.


D offers sound automatic lifetime management? Since when?


Why are you implicitly adding adjectives to my sentence that 
change its meaning?

Re: Garbage Collector?

[Mike Parker via Digitalmars-d]

On Friday, 28 April 2017 at 07:35:00 UTC, Ben wrote:


so I'll reply:
Expanding the continuous memory region a dynamic array 
consists of (possibly moving it) once it overflows has 
absolutely nothing to do with the GC, or even the language, 
it's how the abstract data type "dynamic array" is defined. D 
just does this transparently for you by default. If you 
already know the exact or maximum size, you can allocate 
*once* (not 6 times) using `new` and `.reserve` respectively 
*before* entering the loop, like that article explains in 
depth.


You seem to be missing the fact that i pointed this out. The 
fact that the GC might have done up to 6 collection rounds in 
that loop is "ludicrous".


How is it ludicrous? The fact that you know the GC will only run 
during an allocation, and only if it needs to, allows you to 
control when those opportunities to collect arise. That's a much 
more palatable situation than if it were sitting in the 
background, constantly checking if it needs to collect, then 
doing so without any input from you. There, you'd have no control 
at all.


In the context of an actual program, the example would only have 
made 6 collections if you were putting putting heavy pressure on 
the GC heap, which is an extremely naive way of programming for 
anyone concerned about performance. D allows you several options 
to relieve that pressure and assert some control over when the GC 
can run. Even in a non-GC language, you wouldn't be allocating 
like that in a performance-critical loop -- you would preallocate 
before you entered the loop.

Re: Garbage Collector?

[Ben via Digitalmars-d]

On Thursday, 27 April 2017 at 22:43:56 UTC, Moritz Maxeiner wrote:
You replied to the wrong person here, seeing as I did not link 
to the article you're referring to,


Sorry...


so I'll reply:
Expanding the continuous memory region a dynamic array consists 
of (possibly moving it) once it overflows has absolutely 
nothing to do with the GC, or even the language, it's how the 
abstract data type "dynamic array" is defined. D just does this 
transparently for you by default. If you already know the exact 
or maximum size, you can allocate *once* (not 6 times) using 
`new` and `.reserve` respectively *before* entering the loop, 
like that article explains in depth.


You seem to be missing the fact that i pointed this out. The fact 
that the GC might have done up to 6 collection rounds in that 
loop is "ludicrous".




Um, what? Memory (de)allocation (in C often malloc/free) and 
object (de)contruction (in C usually functions with some naming 
conventions like `type_init`/`type_deinit`) are on two entirely 
different layers! Granted, they are often combined in C to 
functions with names like `type_new`/`type_free`, but they are 
conceptually two distinct things. Just to be very clear, here 
is a primitive diagram of how things work:

make object O of type T:
 --(allocate N bytes)--> [memory chunk M] --(call 
constructor T(args) on M)--> [O]

dispose of O:
[O] --(call destructor ~T() on O)--> [memory chunk M] 
--(deallocate M)--> 


D's garbage collector conceptually changes this to:
make object O of type T:
 --(GC allocates)--> [GC memory pool] --(allocate N 
bytes)--> [memory chunk M] --(call constructor T(args) on M)--> 
[O]

dispose of O:
[O] --(call destructor ~T() on O)--> [memory chunk M] 
--(deallocate M)--> [GC memory pool] --(GC deallocates)--> 
with the specific restriction that you have *no* control over 
'GC deallocates' and only indirect control over 'GC allocates' 
(by allocating more memory from the GC than is available its 
the pool).


Working on the memory chunk layer is memory management.
Working on the object layer is object lifetime management.
D offers you both automatic memory management and automatic 
lifetime management via its GC.
What you describe is manual object lifetime management (which 
is what std.conv.emplace and object.destroy exist for) and has 
no effect on the automatic memory management the GC performs.
You *can* do manual memory management *on top* of the GC's 
memory pool (using core.memory.GC.{alloc/free) or the newer, 
generic Alloactor interface via 
std.experimental.allocator.gc_allocator.GCAllocator.{allocate/deallocate}), but these operations will (generally) not yield any observable difference from the OS's perspective.




That is assuming the GC removes the memory reference when you 
call it. I remember seeing in some other languages ( C# 
possibly? ) that referring a variable to be freed only meant 
the GC freed the memory when it felt like it, not the exact 
spot when you told it.


Again, you seem to mix object lifetime management and memory 
management. You cannot tell the GC to free memory back to the 
operating system (which is what the free syscall does and what 
you seem to be describing). You can *only* free memory you 
allocated *from* the GC *back* to the GC. The GC decides when 
(and if) any memory is ever being freed back to the OS (which 
is kinda one major point of having a GC in the first place).


I know, i do not express myself very well in English.

In my experience most people's aversion to GCs can be aptly 
described by the German proverb "Was der Bauer nicht kennt, das 
frisst er nicht" (the meaning of which being that most people 
generally like living in comfort zones and rarely - if ever - 
dare to leave them of their own free will; that includes 
developers, sadly).


Unfortunately i do have plenty of experience with GC kicking in 
on the wrong moments ( or not at the right moments, people forget 
that one ).


I am sure that the amount of people who develop in GC languages 
is much bigger these days then manual managed languages ( what is 
more or less a rarity these days among the new languages ). Even 
Rust still has some background management going on ( like the 
byte counter ).


Maybe its my opinion only but a good language will not put 
anything in the way of the developer but will point out mistakes.


The Rust compiler is not a bad example but it can be taken a step 
more. Is it so hard for developers when you declare a variable, 
to later also clean it up???


var x = 1;
// Do work
x.free;

Easy ... Sure, it becomes a little bit more tricky with ownership 
but that is where the compiler can help and simply state: "Hey, 
you forgot this variable, it does not seem to be used beyond this 
point". Just calling the x.free seem to be too much work for 
developers these days.


Up to now i found very few languages that did this correctly. But 
this is a offtopic discussion.

Re: Garbage Collector?

[Jerry via Digitalmars-d]

On Thursday, 27 April 2017 at 17:31:42 UTC, bachmeier wrote:

On Thursday, 27 April 2017 at 15:50:56 UTC, Ben wrote:

A few days ago i was reading this topic:

https://news.ycombinator.com/item?id=14165198

And the whole GC keeps coming up as a negative ( compared to 
Rust ).


From my understanding there has been a proposal DIP1000 to 
address this issue. Is there any update on this topic?


Is it possible to run D without the GC AND the standard 
library?


There is opposition to the GC from a small but vocal group of 
mostly C++ developers. Sometimes they even make valid 
arguments. You can find many threads on the topic here.


Citation needed.

Re: Garbage Collector?

[Era Scarecrow via Digitalmars-d]

On Thursday, 27 April 2017 at 19:36:44 UTC, Ben wrote:
Frankly seeing in this example that the GC was in theory able 
to kick in 6 times in a simple 100 item loop, that is not 
efficient. I if did my own memory management, the variable 
cleanup will have been done in one go, right after the loop. 
Simply more efficient.


 I find myself trying to use the stack as much as possible for 
buffers and structs so i can control and throw them away properly 
without worrying about the GC. The downside is they have to be 
fairly small (at least under windows without forcing it using a 
new thread or fiber).

Re: Garbage Collector?

[Ola Fosheim Grostad via Digitalmars-d]

On Thursday, 27 April 2017 at 22:43:56 UTC, Moritz Maxeiner wrote:

Working on the memory chunk layer is memory management.
Working on the object layer is object lifetime management.
D offers you both automatic memory management and automatic 
lifetime management via its GC.


D offers sound automatic lifetime management? Since when?

Re: Garbage Collector?

[Mike Parker via Digitalmars-d]

On Thursday, 27 April 2017 at 19:36:44 UTC, Ben wrote:


Frankly seeing in this example that the GC was in theory able 
to kick in 6 times in a simple 100 item loop, that is not 
efficient. I if did my own memory management, the variable 
cleanup will have been done in one go, right after the loop.


Please reread that bit.

"there were six total GC allocations in the loop. That means 
there were six opportunities for the GC to collect garbage."


Keyword: opportunities. There were six allocations, not six 
collections. The GC only "kicks in" when it needs to reclaim 
memory. In this case, the six allocations are made by the array 
management in DRuntime. It's the same as you allocating with 
`new` six times. In all of those cases, as long as the GC has 
room in its pool from which to allocate, there will be no 
collections.


Preallocate as much as possible and minimize the number of GC 
allocations (allocate on the stack where possible and through 
malloc where necessary) and the GC is not going to get in the way 
for most D programs you write.

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Thursday, 27 April 2017 at 19:36:44 UTC, Ben wrote:
On Thursday, 27 April 2017 at 16:35:57 UTC, Moritz Maxeiner 
wrote:
You'll have to be more specific about what issue you're 
referring to. People not liking garbage collection? In any 
case, AFAIU DIP1000 was about more mechanically verifiable 
memory safety features when not using the GC.




Is it possible to run D without the GC AND the standard 
library?


It is possible to run a D program without the GC only if you 
don't allocate using the GC. If you want to see D code that 
does allocate using the GC, the compiler flag `-vgc` is your 
friend (compile Phobos with `-vgc` for kicks). Currently, not 
all of Phobos is free of GC allocations, most notably 
exceptions (refer to [1] and similar topics)


[1] http://forum.dlang.org/thread/occ9kk$24va$1...@digitalmars.com


Frankly seeing in this example that the GC was in theory able 
to kick in 6 times in a simple 100 item loop, that is not 
efficient. I if did my own memory management, the variable 
cleanup will have been done in one go, right after the loop. 
Simply more efficient.


You replied to the wrong person here, seeing as I did not link to 
the article you're referring to, but your statement about not 
being efficient is, honestly, ludicrous, so I'll reply:
Expanding the continuous memory region a dynamic array consists 
of (possibly moving it) once it overflows has absolutely nothing 
to do with the GC, or even the language, it's how the abstract 
data type "dynamic array" is defined. D just does this 
transparently for you by default. If you already know the exact 
or maximum size, you can allocate *once* (not 6 times) using 
`new` and `.reserve` respectively *before* entering the loop, 
like that article explains in depth.




Been thinking about this topic. Dlang has a destructor, does 
that means if you allocate on the GC and then do your own 
destructor, it technically counts as manual memory management?


Um, what? Memory (de)allocation (in C often malloc/free) and 
object (de)contruction (in C usually functions with some naming 
conventions like `type_init`/`type_deinit`) are on two entirely 
different layers! Granted, they are often combined in C to 
functions with names like `type_new`/`type_free`, but they are 
conceptually two distinct things. Just to be very clear, here is 
a primitive diagram of how things work:

make object O of type T:
 --(allocate N bytes)--> [memory chunk M] --(call constructor 
T(args) on M)--> [O]

dispose of O:
[O] --(call destructor ~T() on O)--> [memory chunk M] 
--(deallocate M)--> 


D's garbage collector conceptually changes this to:
make object O of type T:
 --(GC allocates)--> [GC memory pool] --(allocate N bytes)--> 
[memory chunk M] --(call constructor T(args) on M)--> [O]

dispose of O:
[O] --(call destructor ~T() on O)--> [memory chunk M] 
--(deallocate M)--> [GC memory pool] --(GC deallocates)--> 
with the specific restriction that you have *no* control over 'GC 
deallocates' and only indirect control over 'GC allocates' (by 
allocating more memory from the GC than is available its the 
pool).


Working on the memory chunk layer is memory management.
Working on the object layer is object lifetime management.
D offers you both automatic memory management and automatic 
lifetime management via its GC.
What you describe is manual object lifetime management (which is 
what std.conv.emplace and object.destroy exist for) and has no 
effect on the automatic memory management the GC performs.
You *can* do manual memory management *on top* of the GC's memory 
pool (using core.memory.GC.{alloc/free) or the newer, generic 
Alloactor interface via 
std.experimental.allocator.gc_allocator.GCAllocator.{allocate/deallocate}), but these operations will (generally) not yield any observable difference from the OS's perspective.




That is assuming the GC removes the memory reference when you 
call it. I remember seeing in some other languages ( C# 
possibly? ) that referring a variable to be freed only meant 
the GC freed the memory when it felt like it, not the exact 
spot when you told it.


Again, you seem to mix object lifetime management and memory 
management. You cannot tell the GC to free memory back to the 
operating system (which is what the free syscall does and what 
you seem to be describing). You can *only* free memory you 
allocated *from* the GC *back* to the GC. The GC decides when 
(and if) any memory is ever being freed back to the OS (which is 
kinda one major point of having a GC in the first place).




I personally think that people simple have a bad taste with GC 
because they kick in too much outside there control.


In my experience most people's aversion to GCs can be aptly 
described by the German proverb "Was der Bauer nicht kennt, das 
frisst er nicht" (the meaning of which being that most people 
generally like living in comfort zones and rarely - if ever - 
dare to leave them of their own free will; that includes 
developers, 

Re: Garbage Collector?

[Patric Dexheimer via Digitalmars-d]

GC it´s not only about some performance issues that some people 
may encounter on D, but it its a marketing problem as well.


"D is a systems programming language with..." GC(!?).

Most people that are interest in D came from c/c++ or other 
backgrounds without GC or hearing their entire life that GC "may 
be bad".


Its not a question of "how bad it is" or if it is bad at all.

People know how to program without GC. So this is just something 
in the way.


Anyway D is awesome.

Re: Garbage Collector?

[ag0aep6g via Digitalmars-d]

On Thursday, 27 April 2017 at 19:36:44 UTC, Ben wrote:
Frankly seeing in this example that the GC was in theory able 
to kick in 6 times in a simple 100 item loop, that is not 
efficient. I if did my own memory management, the variable 
cleanup will have been done in one go, right after the loop. 
Simply more efficient.


You can easily disable automatic collections and trigger one 
manually:



import core.memory: GC;
GC.disable(); /* disable automatic collections */
/* ... do your loop ... */
GC.collect(); /* manually trigger collection */
GC.enable(); /* enable automatic collections */


Been thinking about this topic. Dlang has a destructor, does 
that means if you allocate on the GC and then do your own 
destructor, it technically counts as manual memory management?


I'm not sure what role the destructor supposedly plays here. A 
destructor is called automatically when the object is freed, or 
you can call it manually with `object.destroy` [1]. Freeing 
happens when the object is collected, or you can use 
`core.memory.GC.free` [2] to free without collecting. Collecting 
happens automatically, or you can trigger it with 
`core.memory.GC.collect` [3] as shown.


You can use a destructor to manage memory owned by the object. 
But the destructor being there doesn't affect how the object 
itself is managed.


That is assuming the GC removes the memory reference when you 
call it. I remember seeing in some other languages ( C# 
possibly? ) that referring a variable to be freed only meant 
the GC freed the memory when it felt like it, not the exact 
spot when you told it.


Destroying does not imply freeing. It's the other way around. If 
you want to free, call `GC.free`. That does count as manual 
memory management.


I personally think that people simple have a bad taste with GC 
because they kick in too much outside there control. For 90% 
the default behavior is good but its those 10% that leaves a 
bad taste with people ( GC on critical moments and hurting 
performance in return ).


You can:

1) Call `core.memory.GC.disable` [4] if you don't want the GC to 
collect for a while. You should make sure that you don't run out 
of memory during that time, of course. Use 
`core.memory.GC.enable` [5] to enable collections again.


2) Use the `@nogc` [6] attribute to forbid GC allocations in a 
function. GC collections are only triggered by allocations (or 
manually). So the GC won't kick in during execution of @nogc code.




[1] https://dlang.org/phobos/object.html#.destroy
[2] https://dlang.org/phobos/core_memory.html#.GC.free
[3] https://dlang.org/phobos/core_memory.html#.GC.collect
[4] https://dlang.org/phobos/core_memory.html#.GC.disable
[5] https://dlang.org/phobos/core_memory.html#.GC.enable
[6] https://dlang.org/spec/attribute.html#nogc

Re: Garbage Collector?

[Ben via Digitalmars-d]

On Thursday, 27 April 2017 at 16:35:57 UTC, Moritz Maxeiner wrote:
You'll have to be more specific about what issue you're 
referring to. People not liking garbage collection? In any 
case, AFAIU DIP1000 was about more mechanically verifiable 
memory safety features when not using the GC.




Is it possible to run D without the GC AND the standard 
library?


It is possible to run a D program without the GC only if you 
don't allocate using the GC. If you want to see D code that 
does allocate using the GC, the compiler flag `-vgc` is your 
friend (compile Phobos with `-vgc` for kicks). Currently, not 
all of Phobos is free of GC allocations, most notably 
exceptions (refer to [1] and similar topics)


[1] http://forum.dlang.org/thread/occ9kk$24va$1...@digitalmars.com


Frankly seeing in this example that the GC was in theory able to 
kick in 6 times in a simple 100 item loop, that is not efficient. 
I if did my own memory management, the variable cleanup will have 
been done in one go, right after the loop. Simply more efficient.


Been thinking about this topic. Dlang has a destructor, does that 
means if you allocate on the GC and then do your own destructor, 
it technically counts as manual memory management?


That is assuming the GC removes the memory reference when you 
call it. I remember seeing in some other languages ( C# possibly? 
) that referring a variable to be freed only meant the GC freed 
the memory when it felt like it, not the exact spot when you told 
it.


I personally think that people simple have a bad taste with GC 
because they kick in too much outside there control. For 90% the 
default behavior is good but its those 10% that leaves a bad 
taste with people ( GC on critical moments and hurting 
performance in return ).

Re: Garbage Collector?

[bachmeier via Digitalmars-d]

On Thursday, 27 April 2017 at 15:50:56 UTC, Ben wrote:

A few days ago i was reading this topic:

https://news.ycombinator.com/item?id=14165198

And the whole GC keeps coming up as a negative ( compared to 
Rust ).


From my understanding there has been a proposal DIP1000 to 
address this issue. Is there any update on this topic?


Is it possible to run D without the GC AND the standard library?


You might find this blog post relevant:
http://dlang.org/blog/2017/03/20/dont-fear-the-reaper/

There is opposition to the GC from a small but vocal group of 
mostly C++ developers. Sometimes they even make valid arguments. 
You can find many threads on the topic here.

Re: Garbage Collector?

[Moritz Maxeiner via Digitalmars-d]

On Thursday, 27 April 2017 at 15:50:56 UTC, Ben wrote:

A few days ago i was reading this topic:

https://news.ycombinator.com/item?id=14165198

And the whole GC keeps coming up as a negative ( compared to 
Rust ).


That's subjective, at best.
I see most of Rust's ownership mechanics in a negative light 
(love the lisp-inspired syntax, though), as they incur a severe 
productivity decrease while providing little safety benefit for 
/me/ over what I already have in D. But, as always, YMMV.




From my understanding there has been a proposal DIP1000 to 
address this issue.


You'll have to be more specific about what issue you're referring 
to. People not liking garbage collection? In any case, AFAIU 
DIP1000 was about more mechanically verifiable memory safety 
features when not using the GC.




Is it possible to run D without the GC AND the standard library?


It is possible to run a D program without the GC only if you 
don't allocate using the GC. If you want to see D code that does 
allocate using the GC, the compiler flag `-vgc` is your friend 
(compile Phobos with `-vgc` for kicks). Currently, not all of 
Phobos is free of GC allocations, most notably exceptions (refer 
to [1] and similar topics)


[1] http://forum.dlang.org/thread/occ9kk$24va$1...@digitalmars.com

Re: Garbage Collector

[Observer via Digitalmars-d]

On Wednesday, 15 June 2016 at 20:43:55 UTC, deadalnix wrote:
Simple exercise. You have 100 000 servers. Your application 
suddenly become 1% slower. How angry is your CFO when he 
discovers how many new machines he needs to buy ?


Probably not too angry at all.  This is still just a 1% budget
increase, which amounts to a rounding error.  Say those 100K
servers cost $2K each, meaning $200M for the lot.  An extra
$2M capital costs doesn't mean much in that context.  Perhaps
a bigger issue might be the ongoing extra cost for energy,
which applies to all the machines, not just tne new ones.

Look at it another way.  Anyone running 100_000 machines will
certainly not be running them all flat-out, to where a 1%
increase will push out a requirement for more machines.  One
needs extra capacity anyway to handle usual surges in the
volume of business being handled by the servers.

Look at it yet another way.  Sure, $2M is a big number in
absolute terms, for most of us.  But if I were that CFO,
instead of yelling about the problem, I'd go to the CTO
and tell him to take 100 machines out of service and have
the developers use them to profile the application and
find places where much more than 1% can be saved.

Re: Garbage Collector

[dewitt via Digitalmars-d]

On Friday, 17 June 2016 at 21:47:36 UTC, Joerg Joergonson wrote:

On Friday, 17 June 2016 at 04:20:23 UTC, Jonathan Marler wrote:

On Wednesday, 15 June 2016 at 13:38:33 UTC, ketmar wrote:

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:

[...]


you are wrong. and you definitely know nothing about garbage 
collection, virtual machines and code generation. i wonder 
why people keep coming with "suggestions" and "solutions" 
without even a small knowledge in problem field.


That's pretty harsh Ketmar.  It's obvious he knows the general 
ideas and was just wondering if using the .NET GC was a viable 
option.  I think responding to others in such a demeaning way 
is harmful to the D community as it isolates people.  It 
doesn't encourage people to be curious or want to start a 
discussion.  Having people, especially newcomers to D come in 
and make suggestions and solutions is a great thing for a 
community.  It means they saw enough potential in the language 
to want to know more and maybe even how they could contribute.


It also makes ketmar look like a tard and childish. Konstantin 
said he 'believed' something then ketmar responded with a 
fallacious attack. Maybe ketmar needs to take his meds? ;)


The idea that "a community" cannot create a GC is false.  It is 
not too complex as there are plenty of complex projects that are 
community driven.  A better assumption would be he doesn't 
believe THIS community can create a GC.  Not that I believe 
either because there is no reason a community driven GC would not 
be successful. There are plenty of good GCs out there like Java's 
for instance and work great for the ecosystem of the language but 
even if we had the greatest then some D devs would still be upset 
because a lot of D devs come from C/C++ and do not want the GC.   
 Both statements made assumptions and I do not think they are 
even close to the worst things said on this forum.  ppl chill a 
little.  Light one up, Drink one, put some rounds down range, 
whatever you gotta do to lighten up a little.

Re: Garbage Collector

[Joerg Joergonson via Digitalmars-d]

On Friday, 17 June 2016 at 04:20:23 UTC, Jonathan Marler wrote:

On Wednesday, 15 June 2016 at 13:38:33 UTC, ketmar wrote:

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:

[...]


you are wrong. and you definitely know nothing about garbage 
collection, virtual machines and code generation. i wonder why 
people keep coming with "suggestions" and "solutions" without 
even a small knowledge in problem field.


That's pretty harsh Ketmar.  It's obvious he knows the general 
ideas and was just wondering if using the .NET GC was a viable 
option.  I think responding to others in such a demeaning way 
is harmful to the D community as it isolates people.  It 
doesn't encourage people to be curious or want to start a 
discussion.  Having people, especially newcomers to D come in 
and make suggestions and solutions is a great thing for a 
community.  It means they saw enough potential in the language 
to want to know more and maybe even how they could contribute.


It also makes ketmar look like a tard and childish. Konstantin 
said he 'believed' something then ketmar responded with a 
fallacious attack. Maybe ketmar needs to take his meds? ;)

Re: Garbage Collector

[Jonathan Marler via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:38:33 UTC, ketmar wrote:

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:

I don’t believe a community is capable of creating a good GC.


you are wrong. and you definitely know nothing about garbage 
collection, virtual machines and code generation. i wonder why 
people keep coming with "suggestions" and "solutions" without 
even a small knowledge in problem field.


That's pretty harsh Ketmar.  It's obvious he knows the general 
ideas and was just wondering if using the .NET GC was a viable 
option.  I think responding to others in such a demeaning way is 
harmful to the D community as it isolates people.  It doesn't 
encourage people to be curious or want to start a discussion.  
Having people, especially newcomers to D come in and make 
suggestions and solutions is a great thing for a community.  It 
means they saw enough potential in the language to want to know 
more and maybe even how they could contribute.

Re: Garbage Collector

[thedeemon via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:

Has anyone thought about taking GC from .NET and reusing it in 
D?


One significant point has been already mentioned: cost of write 
barriers.
I'd like to mention another factor: .NET GC is a copying one, it 
moves data around. One good feature of current D is it never 
moves data, so you can very easily call C and C++ code and pass 
pointers to your buffers and stuff and C/C++ code just takes 
these pointers and works with them as usual. No pinning, no 
marshaling, zero overhead. If you take a moving GC like .NET's, 
you immediately make all C/C++ interaction much harder, now you 
need to worry about pinning stuff or copying "managed" data to 
"unmanaged" memory and back. This is all costly both in terms of 
CPU cycles and of programmer cycles. You'll need "FFI", what most 
other GC-ed languages have to have, and D doesn't.

Re: Garbage Collector

[rikki cattermole via Digitalmars-d]

On 16/06/2016 6:53 AM, Konstantin wrote:

On Wednesday, 15 June 2016 at 17:02:11 UTC, rikki cattermole wrote:


Higher level languages like Java have the benefit of using pools and
optimizing for this usage pattern, D does and will never have this.

Why don't you want the same for D?


Because we don't need them.
Sprinkling of fairy dust is for stories, not reality.


Well if you really insist to have a String class don't be too
surprised for some reason it doesn't have the same performance to say
Java.

Some areas, like compiling, or producing HTML/XML/JSON documents,
manipulate strings a lot.
Other areas, like GUI editors for sufficiently complex documents, or
level editors for videogame, need to efficiently manipulate huge trees
of assorted small objects, not necessarily strings.


You're quite right and that is why we have a GC to begin with.
Its also part of the reason why std.experimental.allocator will allow 
you to create an allocator that is able to handle such work load and 
then free when complete.



Aka don't go around creating/destroying classes a huge amount unless
you have rolled some form of memory management policy such as
reserving memory for the GC to use.

Yeah, that’s what I regularly do in C++ when I need to efficiently
create/destroys many small objects. Sure, this typically leads to the
best performance, e.g. because I can make the memory layout as cache
friendly as humanly possible.
But not all projects need that. And even for very performance demanding
apps, not all components of the app need that.
For such cases, a good GC (that just works well out of the box like
.NET's GC does) can reduce development costs significantly.


So exactly like what our GC does do. Unless you're doing real time 
development in any form (e.g. sound) you won't need to do much to work 
around the GC.

Re: Garbage Collector

[deadalnix via Digitalmars-d]

On Wednesday, 15 June 2016 at 20:22:21 UTC, Jack Stouffer wrote:

On Wednesday, 15 June 2016 at 19:39:59 UTC, Konstantin wrote:
Well I’m not sure about the 5% (MS says their write barrier 
overhead is comparable to the cost of a simple method call, 
namely 6.4ns: 
https://msdn.microsoft.com/en-us/library/ms973852.aspx), but 
yeah, there’s some tradeoff, for having a good GC.


Even 1% overhead is unacceptable. Again, it's not reasonable 
for a systems language to have people pay for things they're 
not using.


Write barriers will come to D over Walter's dead body.



Simple exercise. You have 100 000 servers. Your application 
suddenly become 1% slower. How angry is your CFO when he 
discovers how many new machines he needs to buy ?

Re: Garbage Collector

[Jack Stouffer via Digitalmars-d]

On Wednesday, 15 June 2016 at 19:39:59 UTC, Konstantin wrote:
Well I’m not sure about the 5% (MS says their write barrier 
overhead is comparable to the cost of a simple method call, 
namely 6.4ns: 
https://msdn.microsoft.com/en-us/library/ms973852.aspx), but 
yeah, there’s some tradeoff, for having a good GC.


Even 1% overhead is unacceptable. Again, it's not reasonable for 
a systems language to have people pay for things they're not 
using.


Write barriers will come to D over Walter's dead body.

By the way, Go implemented those barriers in version 1.5 a year 
ago: https://blog.golang.org/go15gc


Go has no allocation strategies but the GC, so that point is moot.

Re: Garbage Collector

[deadalnix via Digitalmars-d]

On Wednesday, 15 June 2016 at 18:48:28 UTC, jmh530 wrote:
On Wednesday, 15 June 2016 at 18:28:42 UTC, Edwin van Leeuwen 
wrote:


I think he meant that the .NET GC (and most GC designs) rely 
on write barriers, but D does not have write barriers, since D 
is meant to be a proper systems language.


My reading of that LuaJIT GC document is that it requires write 
barriers, but that they are very cheap.


Problem is, in D, many people want to NOT use a GC, and this is 
something we want to support. These people also do NOT want to 
pay for write barrier they do not use.


That being said, we can do write barrier leveraging MMU on 
immutable (it'd be too expensive to do it on mutable data) during 
collection only, so that people that do not want to use the GC do 
not pay for it. The technique is used successfully in the ML 
family of languages for ages now with great results.


Generally, I think the right path forward for D's GC is not to 
emulate managed language's GC as this clearly won't be acceptable 
for many users. On the other hand,

we should:
1/ Leverage D's type system as to get infos about 
mutability/thread locality and segregate the heap accordingly/use 
adapted technique for each.
2/ Make sure the GC can deliver malloc grade performance in an 
alloc/free scenario, as to enable hybrid approach and allow 
people to rely on the GC to the extent they are willing to pay 
for. jemalloc internal datastructures are very amendable to build 
a GC.


I started using this approach in SDC's GC. The only thing 
preventing me to move faster here is simply the time I can 
allocate to solve that problem.

Re: Garbage Collector

[cym13 via Digitalmars-d]

On Wednesday, 15 June 2016 at 19:39:59 UTC, Konstantin wrote:

On Wednesday, 15 June 2016 at 18:23:52 UTC, Jack Stouffer wrote:

They're not acceptable for a systems programming language as 
they require you to pay for something that you might not use.


According to our resident GC maintainer (among many other 
things), they would cause a 1%-5% slow down in the language: 
https://github.com/dlang/druntime/pull/1081#issuecomment-69151660


Well I’m not sure about the 5% (MS says their write barrier 
overhead is comparable to the cost of a simple method call, 
namely 6.4ns: 
https://msdn.microsoft.com/en-us/library/ms973852.aspx), but 
yeah, there’s some tradeoff, for having a good GC.


By the way, Go implemented those barriers in version 1.5 a year 
ago: https://blog.golang.org/go15gc


May I point out that you do not seem to have any kind of 
experience
of D's GC? Try it and see for yourself wether it actually stops 
you
or not. It's right that not everyone is pleased with the current 
GC

but those users have specific expectations and I'm not certain at
all that they would find C# or Go's GC acceptable either.

The point is, D doesn't have to have a GC. Not using it is way 
easier

than in most other languages because all the tools to help you
profile it and avoid it are provided by the compiler. Go without a
good GC is a dead language. D without a good GC is just not as 
good

as it could be. And btw we're generally faster than Go ;-)

The point is: while a better GC is a work in progress we'll 
*never*
have a GC that can fit all needs, but it's not as critical as it 
is

in Go or in C# because we provide other ways to manage memory when
limitations arise.

Re: Garbage Collector

[Konstantin via Digitalmars-d]

On Wednesday, 15 June 2016 at 18:23:52 UTC, Jack Stouffer wrote:

They're not acceptable for a systems programming language as 
they require you to pay for something that you might not use.


According to our resident GC maintainer (among many other 
things), they would cause a 1%-5% slow down in the language: 
https://github.com/dlang/druntime/pull/1081#issuecomment-69151660


Well I’m not sure about the 5% (MS says their write barrier 
overhead is comparable to the cost of a simple method call, 
namely 6.4ns: 
https://msdn.microsoft.com/en-us/library/ms973852.aspx), but 
yeah, there’s some tradeoff, for having a good GC.


By the way, Go implemented those barriers in version 1.5 a year 
ago: https://blog.golang.org/go15gc

Re: Garbage Collector

[ketmar via Digitalmars-d]

On Wednesday, 15 June 2016 at 18:48:28 UTC, jmh530 wrote:
My reading of that LuaJIT GC document is that it requires write 
barriers, but that they are very cheap.


...for language that was originally VM-based. yet they'll have a 
noticable impact on language like D -- especially when programmer 
want to opt-out GC.

Re: Garbage Collector

[Konstantin via Digitalmars-d]

On Wednesday, 15 June 2016 at 17:02:11 UTC, rikki cattermole 
wrote:


Higher level languages like Java have the benefit of using 
pools and optimizing for this usage pattern, D does and will 
never have this.

Why don't you want the same for D?

Well if you really insist to have a String class don't be too 
surprised for some reason it doesn't have the same performance 
to say Java.
Some areas, like compiling, or producing HTML/XML/JSON documents, 
manipulate strings a lot.
Other areas, like GUI editors for sufficiently complex documents, 
or level editors for videogame, need to efficiently manipulate 
huge trees of assorted small objects, not necessarily strings.


Aka don't go around creating/destroying classes a huge amount 
unless you have rolled some form of memory management policy 
such as reserving memory for the GC to use.
Yeah, that’s what I regularly do in C++ when I need to 
efficiently create/destroys many small objects. Sure, this 
typically leads to the best performance, e.g. because I can make 
the memory layout as cache friendly as humanly possible.
But not all projects need that. And even for very performance 
demanding apps, not all components of the app need that.
For such cases, a good GC (that just works well out of the box 
like .NET's GC does) can reduce development costs significantly.

Re: Garbage Collector

[jmh530 via Digitalmars-d]

On Wednesday, 15 June 2016 at 18:28:42 UTC, Edwin van Leeuwen 
wrote:


I think he meant that the .NET GC (and most GC designs) rely on 
write barriers, but D does not have write barriers, since D is 
meant to be a proper systems language.


My reading of that LuaJIT GC document is that it requires write 
barriers, but that they are very cheap.

Re: Garbage Collector

[Edwin van Leeuwen via Digitalmars-d]

On Wednesday, 15 June 2016 at 17:03:21 UTC, Konstantin wrote:

On Wednesday, 15 June 2016 at 13:56:09 UTC, Jack Stouffer wrote:
Has anyone thought about taking GC from .NET and reusing it 
in D?

Two words: write barriers.
What about them? You mean not all D’s target platforms support 
them?


I think he meant that the .NET GC (and most GC designs) rely on 
write barriers, but D does not have write barriers, since D is 
meant to be a proper systems language.

Re: Garbage Collector

[Jack Stouffer via Digitalmars-d]

On Wednesday, 15 June 2016 at 17:03:21 UTC, Konstantin wrote:

Two words: write barriers.


What about them? You mean not all D’s target platforms support 
them?


They're not acceptable for a systems programming language as they 
require you to pay for something that you might not use.


According to our resident GC maintainer (among many other 
things), they would cause a 1%-5% slow down in the language: 
https://github.com/dlang/druntime/pull/1081#issuecomment-69151660

Re: Garbage Collector

[Konstantin via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:56:09 UTC, Jack Stouffer wrote:

One guy wrote the LuaJIT GC, which beat almost everyone else in 
performance when I last checked
“The current garbage collector is relatively slow compared to 
implementations for other language runtimes. It's not competitive 
with top-of-the-line GCs, especially for large workloads.“

https://github.com/LuaJIT/LuaJIT/issues/38

They have planned something for 3.0 that may or may not work:
http://wiki.luajit.org/New-Garbage-Collector
But that’s merely a design, AFAIK there’s no implementation. 
They’re still looking for a sponsor for that.


Has anyone thought about taking GC from .NET and reusing it in 
D?

Two words: write barriers.
What about them? You mean not all D’s target platforms support 
them?

Re: Garbage Collector

[rikki cattermole via Digitalmars-d]

On 16/06/2016 4:52 AM, Konstantin wrote:

On Wednesday, 15 June 2016 at 13:40:11 UTC, rikki cattermole wrote:


5. The requirements for our GC is quite intricate. I.e. you can't just
pop in one that doesn't understand about our Thread Local Storage (TLS)
and stuff.

D’s TLS that different from .NET's TLS?
https://msdn.microsoft.com/en-us/library/system.threadstaticattribute(v=vs.110).aspx


Yes it most definitely is.
We roll our own for platforms that do not support it.

There is an abstraction in druntime specifically to handle this problem.


I forgot to mention, good D code is not the same as a higher level
language like Java.
Here you don't have the automagick behavior of arrays. If you append
it will have a high cost. All allocations have a large cost. Instead
allocate in one large block which will of course be a whole lot faster
then small tiny ones.

You’re saying memory allocations in D are generally very expensive, but
that’s not a problem, because it already functions as designed?


No.
Memory allocations are /always/ expensive.

Higher level languages like Java have the benefit of using pools and 
optimizing for this usage pattern, D does and will never have this.


Keep in mind an allocation = usage of malloc + write to returned pointer.


So even if the GC is enabled, good D code won't cause too much slow
down unless you decide to write heavy OOP code.

I’ve been developing heavy OOP code in various languages (mostly C++,
but also C# and Objective-C) for decades already. OOP works very well
for me.


Well if you really insist to have a String class don't be too surprised 
for some reason it doesn't have the same performance to say Java.
Aka don't go around creating/destroying classes a huge amount unless you 
have rolled some form of memory management policy such as reserving 
memory for the GC to use.


We have other tools where OOP normally would be used such as templates, 
structs, function pointers and delegates.

Re: Garbage Collector

[Konstantin via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:40:11 UTC, rikki cattermole 
wrote:


5. The requirements for our GC is quite intricate. I.e. you 
can't just
pop in one that doesn't understand about our Thread Local 
Storage (TLS)

and stuff.
D’s TLS that different from .NET's TLS? 
https://msdn.microsoft.com/en-us/library/system.threadstaticattribute(v=vs.110).aspx


I forgot to mention, good D code is not the same as a higher 
level language like Java.
Here you don't have the automagick behavior of arrays. If you 
append it will have a high cost. All allocations have a large 
cost. Instead allocate in one large block which will of course 
be a whole lot faster then small tiny ones.
You’re saying memory allocations in D are generally very 
expensive, but that’s not a problem, because it already functions 
as designed?


So even if the GC is enabled, good D code won't cause too much 
slow down unless you decide to write heavy OOP code.
I’ve been developing heavy OOP code in various languages (mostly 
C++, but also C# and Objective-C) for decades already. OOP works 
very well for me.

Re: Garbage Collector

[Konstantin via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:27:47 UTC, jmh530 wrote:


Possible to disable.
I don’t want to: for the last couple years I’ve been developing 
50/50 C++/C#, and I admit I’m usually more productive using C#, 
one of the reasons of that is GC.


They've got a GSOC guy workin' on it now. I would hold off 
judgements until that effort is concluded.

OK let's see.

Re: Garbage Collector

[Jack Stouffer via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:
I don’t believe a community is capable of creating a good GC. 
It’s just too complex engineering task. It’s been a known 
problem for years, still no solution.


GCs are a solved problem and the most common and fastest 
techniques have been known for more than 20 years. The GC 
implementation that D is using now came from the 70's, for 
example.


One guy wrote the LuaJIT GC, which beat almost everyone else in 
performance when I last checked, so I think this is a massive 
exaggeration.


Has anyone thought about taking GC from .NET and reusing it in 
D?


Two words: write barriers.

Re: Garbage Collector

[rikki cattermole via Digitalmars-d]

On 16/06/2016 1:33 AM, rikki cattermole wrote:

I'm not sure how much you know about D but:

1. Somebody is working on improving D's GC as part of GSOC in the hopes
of making it able to be precise (from memory not 100% sure).
2. Only a few language features forces you to use the GC.
3. For most uses you are not forced to use the GC in any form especially
with the help of std.experimental.allocator.
4. Our GC is based upon the Boehm GC. Its old. Even the more recent
versions would be far better then what we have (we forked a long time ago).
5. The requirements for our GC is quite intricate. I.e. you can't just
pop in one that doesn't understand about our Thread Local Storage (TLS)
and stuff.
6. As said by somebody else we can disable the GC so it won't go ahead
and scan upon allocation (only time it does).


I forgot to mention, good D code is not the same as a higher level 
language like Java.


Here you don't have the automagick behavior of arrays. If you append it 
will have a high cost. All allocations have a large cost. Instead 
allocate in one large block which will of course be a whole lot faster 
then small tiny ones.


So even if the GC is enabled, good D code won't cause too much slow down 
unless you decide to write heavy OOP code.

Re: Garbage Collector

[ketmar via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:

I don’t believe a community is capable of creating a good GC.


you are wrong. and you definitely know nothing about garbage 
collection, virtual machines and code generation. i wonder why 
people keep coming with "suggestions" and "solutions" without 
even a small knowledge in problem field.

Re: Garbage Collector

[Kagamin via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:
Has anyone thought about taking GC from .NET and reusing it in 
D?


Fast GC for D was considered and rejected. What can be done is a 
precise and concurrent GC.

Re: Garbage Collector

[rikki cattermole via Digitalmars-d]

I'm not sure how much you know about D but:

1. Somebody is working on improving D's GC as part of GSOC in the hopes 
of making it able to be precise (from memory not 100% sure).

2. Only a few language features forces you to use the GC.
3. For most uses you are not forced to use the GC in any form especially 
with the help of std.experimental.allocator.
4. Our GC is based upon the Boehm GC. Its old. Even the more recent 
versions would be far better then what we have (we forked a long time ago).
5. The requirements for our GC is quite intricate. I.e. you can't just 
pop in one that doesn't understand about our Thread Local Storage (TLS) 
and stuff.
6. As said by somebody else we can disable the GC so it won't go ahead 
and scan upon allocation (only time it does).

Re: Garbage Collector

[jmh530 via Digitalmars-d]

On Wednesday, 15 June 2016 at 13:19:31 UTC, Konstantin wrote:
Started learning D. Like the language. However, found several 
people complaining about garbage collector’s reliability and 
performance. For me it’s a showstopper.




Possible to disable.

I don’t believe a community is capable of creating a good GC. 
It’s just too complex engineering task. It’s been a known 
problem for years, still no solution.




They've got a GSOC guy workin' on it now. I would hold off 
judgements until that effort is concluded.




Has anyone thought about taking GC from .NET and reusing it in 
D?




I don't think this would work, but I don't know enough to be able 
to explain why.

Re: Garbage Collector : Ignoring a reference

[Guillaume Piolat via Digitalmars-d-learn]

On Tuesday, 26 April 2016 at 09:07:59 UTC, Begah wrote:
I am trying to create an asset manager for my textures. I had 
the idea ( it may be a wrong idea ) to create a hashmap of my 
textures with a string as the key. When the program request a 
texture, it firts check if it is in the hashmap and then 
returns if it is :


Texture[string] textures;

Texture loadTexture(string filename) {
  if(filename in textures)
return textures[filename]
  else
// Load image and put it in hashmap
}

Warning : I haven't tested if it actually doesn't work, but 
thinking about it, i think it should not.
My problem is that i return a reference of the texture to the 
program, but i keep one to myself and i want to free the 
texture if my program isn't using it anymore ( no more 
reference to it ). The problem i see, is that i will always 
have atleast one reference to the texture in my hashmap, but i 
want the garbage collector to ignore that reference and to free 
the texture if there are no more references anywhere in my 
program ( except in the hashmap ).


How could i tell the garbage collector to ignore the reference 
in the hashmap and to free it if there isn't any other 
reference that in my hashmap?


You should avoid to rely on the garbage collector to free 
non-memory resources anyway. This is accidental correctness. You 
are not guaranteed to get called by the GC, or by the appropriate 
thread.


You can call .destroy() on these textures when in the texture 
manager destructor.

Re: Garbage Collector : Ignoring a reference

[Namespace via Digitalmars-d-learn]

On Tuesday, 26 April 2016 at 09:07:59 UTC, Begah wrote:
I am trying to create an asset manager for my textures. I had 
the idea ( it may be a wrong idea ) to create a hashmap of my 
textures with a string as the key. When the program request a 
texture, it firts check if it is in the hashmap and then 
returns if it is :


Texture[string] textures;

Texture loadTexture(string filename) {
  if(filename in textures)
return textures[filename]
  else
// Load image and put it in hashmap
}

Warning : I haven't tested if it actually doesn't work, but 
thinking about it, i think it should not.
My problem is that i return a reference of the texture to the 
program, but i keep one to myself and i want to free the 
texture if my program isn't using it anymore ( no more 
reference to it ). The problem i see, is that i will always 
have atleast one reference to the texture in my hashmap, but i 
want the garbage collector to ignore that reference and to free 
the texture if there are no more references anywhere in my 
program ( except in the hashmap ).


How could i tell the garbage collector to ignore the reference 
in the hashmap and to free it if there isn't any other 
reference that in my hashmap?


Texture[string] textures;

Texture* loadTexture(string filename) {
  if(filename in textures)
return [filename]
  else
// Load image and put it in hashmap
}

Texture* tex = loadTexture(...);

Re: Garbage Collector : Ignoring a reference

[Marco Leise via Digitalmars-d-learn]

Am Tue, 26 Apr 2016 13:35:37 +
schrieb Begah :

> When the screen switches to another screen ie from menu to the 
> game,
> I want that the "button.png" texture is automaticly destroyed by 
> the gc.

My ideological point of view is that you must not use
non-deterministic garbage collection for resources. Neither
for files, GUI widgets or textures. The garbage collector
cannot look past its own confined environment (heap memory
allocated by the D program and loaded D libraries) and will
not have enough information to tell if the process is running
out of file handles, backing buffer for widgets or texture
memory on the graphics card. It only takes action, when its own
heap is filling up or when you manually call GC.collect().
All the convenient 100% GC languages from Java to Python
require the user to call ".close()" for files,
".dispose()/.Destroy()" for widgets, etc.
Reference counting is the correct approach. It makes using
external resources safe and convenient by removing the need
for manual lifetime management. Cyclic references cannot
exist in the D code in this scenario.

The texture constructor:
- sets ref count to 1
- adds texture to hash map
The copy constructor:
- increments the ref count
The destructor:
- decrements the ref count
- if ref count reaches 0:
  - removes the texture from the hash table
  - destroys the texture data

-- 
Marco

Re: Garbage Collector : Ignoring a reference

[ciechowoj via Digitalmars-d-learn]

Thus, i need a way to tell the gc to ignore the reference ( or 
something similar ) in that hashmap.


So, having pointer that doesn't hold a reference isn't that hard 
(store it in memory region that is unreachable to gc), but don't 
you need a way to tell if that pointer ins't dangling, beyond 
initial problem?

Re: Garbage Collector : Ignoring a reference

[ag0aep6g via Digitalmars-d-learn]

On 26.04.2016 15:35, Begah wrote:

Nothing will reload.

An example :
I load a texture "button.png" in a class and draw it to the screen,
When the screen switches to another screen ie from menu to the game,
I want that the "button.png" texture is automaticly destroyed by the gc.
But this will never happen because i still have a reference to it in my
hashmap.
Thus, i need a way to tell the gc to ignore the reference ( or something
similar ) in that hashmap.


How would you prevent reads of that now-invalid element of the hashmap?

Re: Garbage Collector : Ignoring a reference

[Begah via Digitalmars-d-learn]

On Tuesday, 26 April 2016 at 13:01:26 UTC, ciechowoj wrote:

On Tuesday, 26 April 2016 at 09:07:59 UTC, Begah wrote:
How could i tell the garbage collector to ignore the reference 
in the hashmap and to free it if there isn't any other 
reference that in my hashmap?


You could always zero the reference in the hashmap, as it won't 
be valid after reload anyway...


Nothing will reload.

An example :
I load a texture "button.png" in a class and draw it to the 
screen,
When the screen switches to another screen ie from menu to the 
game,
I want that the "button.png" texture is automaticly destroyed by 
the gc.
But this will never happen because i still have a reference to it 
in my hashmap.
Thus, i need a way to tell the gc to ignore the reference ( or 
something similar ) in that hashmap.

Re: Garbage Collector : Ignoring a reference

[ciechowoj via Digitalmars-d-learn]

On Tuesday, 26 April 2016 at 09:07:59 UTC, Begah wrote:
How could i tell the garbage collector to ignore the reference 
in the hashmap and to free it if there isn't any other 
reference that in my hashmap?


You could always zero the reference in the hashmap, as it won't 
be valid after reload anyway...

Re: Garbage Collector : Ignoring a reference

[Alex Parrill via Digitalmars-d-learn]

On Tuesday, 26 April 2016 at 09:07:59 UTC, Begah wrote:
I am trying to create an asset manager for my textures. I had 
the idea ( it may be a wrong idea ) to create a hashmap of my 
textures with a string as the key. When the program request a 
texture, it firts check if it is in the hashmap and then 
returns if it is :


[...]


What you want are "weak references". I don't think D supports 
them yet.

Re: Garbage collector collects live objects

[Ali Çehreli via Digitalmars-d-learn]

On 12/09/2014 08:53 AM, Steven Schveighoffer wrote:

 On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:

 that file can be already finalized. please remember that `~this()` is
 more a finalizer than destructor, and it's called on *dead* object.

Agreed: D has a terminology issue here, what we call a class 
destructor is a class finalizer. I have added D to the Wikipedia article:


  http://en.wikipedia.org/wiki/Finalizer

Now I want to improve some of my chapters.

 here this means that any other object in your object (including
 structs) can be already finalized at the time GC decides to call your
 finalizer.

Although I know the fact above, I think ketmar makes a distinction (that 
is new to me) that a finalizer is a function that does not reference 
class members but a destructor does (or safely can).


 File is specially designed (although it's not perfect) to be able to
 close in the GC. Its ref-counted payload is placed on the C heap to
 allow access during finalization.

 That being said, you actually don't need to write the above in the class
 finalizer, _file's destructor will automatically be called.

 just avoid destructors unless you *really* need that. in your case
 simply let GC finalize your File, don't try to help GC. this is not C++
 (or any other language without GC) and destructors aren't destructing
 anything at all. destructors must clean up the things that GC cannot
 (malloc()'ed memory, for example), and nothing else.


 Good advice ;)

 I would say other than library writers, nobody should ever write a class
 dtor.

 -Steve

Can somebody elaborate on that guideline please. Given a case where 
runtime polymorphism is needed, so that we have to use classes, does the 
guideline simply mean that arrange for manual cleanup or is there more 
in that guideline? I am confused about the library writers part. :)


Thank you,
Ali

Re: Garbage collector collects live objects

[Steven Schveighoffer via Digitalmars-d-learn]

On 3/25/15 9:51 AM, Ali Çehreli wrote:

On 12/09/2014 08:53 AM, Steven Schveighoffer wrote:

  On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:

  that file can be already finalized. please remember that `~this()` is
  more a finalizer than destructor, and it's called on *dead* object.

Agreed: D has a terminology issue here, what we call a class
destructor is a class finalizer. I have added D to the Wikipedia
article:

   http://en.wikipedia.org/wiki/Finalizer

Now I want to improve some of my chapters.

  here this means that any other object in your object (including
  structs) can be already finalized at the time GC decides to call your
  finalizer.

Although I know the fact above, I think ketmar makes a distinction (that
is new to me) that a finalizer is a function that does not reference
class members but a destructor does (or safely can).


Think of it this way -- a D destructor can access data from it's own 
object. This means it CAN access its own members that are structs or 
value types. It should NOT access referenced data, unless that data is 
allocated outside the GC.


In Tango, there was a finalizer and a destructor. The finalizer was 
called from the GC. The destructor was called when using the 'delete' 
operator explicitly (and I think it also called the finalizer 
afterward). Having this informational structure helps to know where to 
put what cleanup code. I would love it if druntime added this feature, 
or something similar.




  File is specially designed (although it's not perfect) to be able to
  close in the GC. Its ref-counted payload is placed on the C heap to
  allow access during finalization.
 
  That being said, you actually don't need to write the above in the class
  finalizer, _file's destructor will automatically be called.
 
  just avoid destructors unless you *really* need that. in your case
  simply let GC finalize your File, don't try to help GC. this is not C++
  (or any other language without GC) and destructors aren't destructing
  anything at all. destructors must clean up the things that GC cannot
  (malloc()'ed memory, for example), and nothing else.
 
 
  Good advice ;)
 
  I would say other than library writers, nobody should ever write a class
  dtor.
 
  -Steve

Can somebody elaborate on that guideline please. Given a case where
runtime polymorphism is needed, so that we have to use classes, does the
guideline simply mean that arrange for manual cleanup or is there more
in that guideline? I am confused about the library writers part. :)


Destructors should only be used to clean non-GC resources (and in terms 
of completeness, you should implement such a destructor). It is 
important to remember that the GC may NEVER get around to calling your 
destructor. This means you cannot depend on it releasing non-GC 
resources in a timely manner.


A good example is a file descriptor. If you have an object that contains 
a file descriptor you SHOULD have a destructor which closes the file 
descriptor if not already closed. However, you should NOT technically 
rely on it being called in a timely manner. For cases where you can do 
it synchronously, use a specific method to close the file descriptor.


There is a school of thought that says it is an error to rely on the GC 
to destroy the FD, so why even have the destructor. But I see no reason 
to leak the FD out of spite.


This is why I say it should be only library writers -- they are the ones 
creating wrapper objects and using low level operations.


Of course, this isn't a hard and fast rule.

-Steve

Re: Garbage collector returning pointers

[Robert M. Münch via Digitalmars-d-learn]

On 2015-03-15 15:57:43 +, Marc Schütz said:

Ok. I need to dig into how the interpreter handles the returned pointer 
and how the stack is handled.


C usually uses manual memory management, therefore I would expect that 
the interpreter actually documents whom the pointer belongs to, and who 
is responsible for cleaning it up.


What I'm trying to understand is how the returned pointer is used and 
when the GC can kick in.


I tracked the return code from the DLL. The returned pointer is 
returned in EAX register (conforming to the C calling convention). Then 
the interpreter uses this pointer for some time (in my case to create a 
copy).


It than continues, and at some point the EAX register gets some other 
value. This is the point, wher no reference to the D GC allocated 
memory can anylonger be found. Hence the GC should be free to free the 
memory.


If the interpreter takes ownership, you should just allocate the data 
with malloc(), and the interpreter will then call free() on it when it 
doesn't need it any longer. Not sure whether .dup is compatible with 
that, maybe you'd need to write a small helper that copies things to 
the C heap.


Yes, exactly. And the problem is, that it's not clear to me (yet) when 
the interpreter takes ownership and when not.


Am I right, that I can query() the D GC to check if a pointer is known 
/ can be found or not?


It would be cool if I could query D from the interpreter and ask: Will 
this pointer be freed in the next collector run? and get back: Yes, or 
already has been.


--
Robert M. Münch
http://www.saphirion.com
smarter | better | faster

Re: Garbage collector returning pointers

[Robert M. Münch via Digitalmars-d-learn]

On 2015-03-14 20:45:21 +, Marc Schütz said:

As long as the pointer remains on the stack or in a register, the GC 
will keep the allocation alive.


Hi Marc, ok, got it. Thanks.

 But if your C code stores the pointer on the C heap or in a global, 
the GC won't know anything about it and can free it.


Ok. I need to dig into how the interpreter handles the returned pointer 
and how the stack is handled.


--
Robert M. Münch
http://www.saphirion.com
smarter | better | faster

Re: Garbage collector returning pointers

[via Digitalmars-d-learn]

On Sunday, 15 March 2015 at 15:08:43 UTC, Robert M. Münch wrote:

On 2015-03-14 20:45:21 +, Marc Schütz said:

As long as the pointer remains on the stack or in a register, 
the GC will keep the allocation alive.


Hi Marc, ok, got it. Thanks.

But if your C code stores the pointer on the C heap or in a 
global, the GC won't know anything about it and can free it.


Ok. I need to dig into how the interpreter handles the returned 
pointer and how the stack is handled.


C usually uses manual memory management, therefore I would expect 
that the interpreter actually documents whom the pointer belongs 
to, and who is responsible for cleaning it up.


If the interpreter takes ownership, you should just allocate the 
data with malloc(), and the interpreter will then call free() on 
it when it doesn't need it any longer. Not sure whether .dup is 
compatible with that, maybe you'd need to write a small helper 
that copies things to the C heap.


Otherwise, maybe the interpreter will call you back when it wants 
to free the memory.


In both cases, you don't need to use GC pointers at all, because 
it's actually doing manual memory management.

Re: Garbage collector returning pointers

[via Digitalmars-d-learn]

On Saturday, 14 March 2015 at 18:26:34 UTC, Robert M. Münch wrote:

Hi, I have a question about how the GC handles this case:

export extern(C) char* foo(){
 char[] x = This is a dynamic D string..dup;

 return(cast(char*)x);
}

Since x is pointer to array data  length if it goes out of 
scope, it's destroyed and the last reference to the array data 
is gone. Hence, the GC could kick in and free the array data. 
Is this correct?


Or will the GC know, that there was a pointer to the array data 
returned and hence a new reference exists as long until someone 
tells the GC that the pointer is no longer used?


My situation is, that the returned pointer is used to copy the 
result to some interpreter internal state. Depending on the 
answers above, there could be a short time where the memory 
state is collectable before the coyping was finished.


As long as the pointer remains on the stack or in a register, the 
GC will keep the allocation alive. But if your C code stores the 
pointer on the C heap or in a global, the GC won't know anything 
about it and can free it.

Re: Garbage collector returning pointers

[anonymous via Digitalmars-d-learn]

On Saturday, 14 March 2015 at 18:26:34 UTC, Robert M. Münch wrote:

Hi, I have a question about how the GC handles this case:

export extern(C) char* foo(){
 char[] x = This is a dynamic D string..dup;

 return(cast(char*)x);
}


Returning `x.ptr` would look a little nicer.



Since x is pointer to array data  length if it goes out of 
scope, it's destroyed and the last reference to the array data 
is gone. Hence, the GC could kick in and free the array data. 
Is this correct?


No.

Or will the GC know, that there was a pointer to the array data 
returned and hence a new reference exists as long until someone 
tells the GC that the pointer is no longer used?


Yes. The returned pointer is a reference. Once that reference is 
gone, the GC can collect the array. You don't need to explicitly 
inform the GC when you're done with the pointer.


My situation is, that the returned pointer is used to copy the 
result to some interpreter internal state. Depending on the 
answers above, there could be a short time where the memory 
state is collectable before the coyping was finished.


I think you're safe.

Re: Garbage collector collects live objects

[Steven Schveighoffer via Digitalmars-d-learn]

On 12/12/14 10:50 AM, Steven Schveighoffer wrote:

On 12/12/14 7:52 AM, Ruslan Mullakhmetov wrote:



btw, i used suggested trackallocs.d and GC defenetely receives NO_SCAN

before tag: 1 len: 2 ptr: 103A78058 root: 103A77000:8192 attr: APPENDABLE
gc_qalloc(41, NO_SCAN APPENDABLE ) cc: 29106 asz: 10152603, ti: null
ret: BlkInfo_(104423800, 64, 10)
after tag: 1 len: 3 ptr: 104423810 root: 104423800:64 attr: NO_SCAN
APPENDABLE


This is good information, thanks. I will get back to you with a druntime
branch to try. Can I email you at this address? If not, email me at the
address from my post to let me know your contact, no reason to work
through building issues on the public forum :)


For those who were interested, we were not able to solve this problem, 
and unfortunately Ruslan's company cannot keep trying to debug, they 
have moved on to another language :(


If anyone else has this kind of issue (array appending causing flags to 
change), please let me know, I would like to make sure this gets solved.


I am creating 2 fixes, one to fix the issue with the offset (in 
progress) and after that I will attempt to ensure any changes to a 
block's flags stick when the array is appended (I found at least one 
place in Phobos where this can cause a bug, but it's not affecting his 
code).


Ruslan, you may want to try the second fix when it has been added just 
to see if it helps.


The two bug reports are:

https://issues.dlang.org/show_bug.cgi?id=13854
https://issues.dlang.org/show_bug.cgi?id=13878

-Steve

Re: Garbage collector collects live objects

[Ruslan Mullakhmetov via Digitalmars-d-learn]

On Thursday, 11 December 2014 at 18:36:59 UTC, Steven 
Schveighoffer wrote:

My analysis so far:

2. In the array append code, the block attributes are obtained 
via GC.query, which has this code for getting the attributes:


https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1792

Quoting from that function:

// reset the offset to the base pointer, otherwise the bits
// are the bits for the pointer, which may be garbage
offset = cast(size_t)(info.base - pool.baseAddr);
info.attr = getBits(pool, cast(size_t)(offset  pool.shiftBy));

Which should get the correct bits. I suspected there was an 
issue with getting the wrong bits, but this code looks correct.


3. The runtime caches the block info for thread local data for 
append speed. A potential issue is that the attributes are 
cached from a previous use for that block, but the GC (and the 
runtime itself) SHOULD clear that cache entry when that block 
is freed, avoiding this issue. A potential way to check this is 
to assert in a debug build of druntime that the cached block 
info always equals the actual block info. Are you able to build 
a debug version of druntime to test this? I can give you the 
changes you should make. This would explain the great 
difficulty in reproducing the issue.


I will try to build debug version of dmd compiler and check the 
issue.




4. If your code is multi-threaded, but using __gshared, it can 
make the cache incorrect. Are you doing this?




the app is multi-threaded via std.concurrency.

there is only one known to me place where __gshared is used: 
logging library (checked by searching through whole source tree). 
make stub for this lib and try, so identify whether cache 
invalidated by _gshared or not.


But the cache is really the only possible place I can see where 
the bits are set incorrectly, given that you just verified the 
bits are correct before the append.


Can you just list the version of the compiler you are using? I 
want to make sure this isn't an issue that has already been 
fixed.


the last. first of all i updated whole toolchain (dmd, dub).

$ dmd
DMD64 D Compiler v2.066.1


-Steve


I started looking druntime and dmd source code myself before i 
checked the thread (thsnks for your help and feedback) and i have 
some questions. could you explain to me something?


i_m looking here 
https://github.com/D-Programming-Language/druntime/blob/v2.066.1/src/rt/lifetime.d#L591


---
line #603
auto size = ti.next.tsize;

why `next`? it can be even null if this is last TypeInfo in the 
linked list.


-

btw, i used suggested trackallocs.d and GC defenetely receives 
NO_SCAN


before tag: 1 len: 2 ptr: 103A78058 root: 103A77000:8192 attr: 
APPENDABLE
gc_qalloc(41, NO_SCAN APPENDABLE ) cc: 29106 asz: 10152603, ti: 
null ret: BlkInfo_(104423800, 64, 10)
after tag: 1 len: 3 ptr: 104423810 root: 104423800:64 attr: 
NO_SCAN APPENDABLE

Re: Garbage collector collects live objects

[Ruslan Mullakhmetov via Digitalmars-d-learn]

On Friday, 12 December 2014 at 12:53:00 UTC, Ruslan Mullakhmetov 
wrote:
On Thursday, 11 December 2014 at 18:36:59 UTC, Steven 
Schveighoffer wrote:

My analysis so far:

4. If your code is multi-threaded, but using __gshared, it can 
make the cache incorrect. Are you doing this?




the app is multi-threaded via std.concurrency.

there is only one known to me place where __gshared is used: 
logging library (checked by searching through whole source 
tree). make stub for this lib and try, so identify whether 
cache invalidated by _gshared or not.





removing __gshared seems does not helped.

Re: Garbage collector collects live objects

[Steven Schveighoffer via Digitalmars-d-learn]

On 12/12/14 7:52 AM, Ruslan Mullakhmetov wrote:

On Thursday, 11 December 2014 at 18:36:59 UTC, Steven Schveighoffer wrote:

My analysis so far:

2. In the array append code, the block attributes are obtained via
GC.query, which has this code for getting the attributes:

https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1792


Quoting from that function:

// reset the offset to the base pointer, otherwise the bits
// are the bits for the pointer, which may be garbage
offset = cast(size_t)(info.base - pool.baseAddr);
info.attr = getBits(pool, cast(size_t)(offset  pool.shiftBy));

Which should get the correct bits. I suspected there was an issue with
getting the wrong bits, but this code looks correct.

3. The runtime caches the block info for thread local data for append
speed. A potential issue is that the attributes are cached from a
previous use for that block, but the GC (and the runtime itself)
SHOULD clear that cache entry when that block is freed, avoiding this
issue. A potential way to check this is to assert in a debug build of
druntime that the cached block info always equals the actual block
info. Are you able to build a debug version of druntime to test this?
I can give you the changes you should make. This would explain the
great difficulty in reproducing the issue.


I will try to build debug version of dmd compiler and check the issue.


A debug version of compiler is not necessary, not even a debug version 
of phobos, just druntime. But it's not going to matter yet, because I 
need to give you the asserts to put in there. I just wanted to know if 
you needed help doing it.






4. If your code is multi-threaded, but using __gshared, it can make
the cache incorrect. Are you doing this?



the app is multi-threaded via std.concurrency.


This should be OK, you should not be able to share data that is not 
marked as shared.



there is only one known to me place where __gshared is used: logging
library (checked by searching through whole source tree). make stub for
this lib and try, so identify whether cache invalidated by _gshared or not.


Here is where it might occur:

1. Due to shared data having typeinfo attached to it that it is actually 
shared, the runtime takes advantage of that. We can use a lock-free 
cache that is thread-local for anything not marked as shared, because 
nothing outside the thread can access that data.
2. __gshared gets around this because it is not marked as shared by the 
compiler. This means, if you, for instance, appended to a __gshared 
array, the runtime would treat it like a thread-local array. If you did 
this from multiple threads, the cache may be invalid in one or more of them.
3. Actual 'shared' arrays are not permitted to use the cache, so they 
should not have this issue.


I see that you removed the only instance of __gshared and it did not 
help. That at least rules that out.



But the cache is really the only possible place I can see where the
bits are set incorrectly, given that you just verified the bits are
correct before the append.

Can you just list the version of the compiler you are using? I want to
make sure this isn't an issue that has already been fixed.


the last. first of all i updated whole toolchain (dmd, dub).

$ dmd
DMD64 D Compiler v2.066.1


Thanks, this at least gives me a baseline to know what to test and debug 
with. I do not believe the code has had any significant fixes that would 
help with this issue since then.




I started looking druntime and dmd source code myself before i checked
the thread (thsnks for your help and feedback) and i have some
questions. could you explain to me something?

i_m looking here
https://github.com/D-Programming-Language/druntime/blob/v2.066.1/src/rt/lifetime.d#L591


---
line #603
auto size = ti.next.tsize;

why `next`? it can be even null if this is last TypeInfo in the linked
list.


This is the way the compiler constructs the type info. The first 
TypeInfo is always TypeInfo_Array (or TypeInfo_Shared, or const or 
whatever), and the .next is the typeinfo for the element type. all this 
does is get the size of an element. Since we know we are dealing with an 
array, we know next is always valid.




btw, i used suggested trackallocs.d and GC defenetely receives NO_SCAN

before tag: 1 len: 2 ptr: 103A78058 root: 103A77000:8192 attr: APPENDABLE
gc_qalloc(41, NO_SCAN APPENDABLE ) cc: 29106 asz: 10152603, ti: null
ret: BlkInfo_(104423800, 64, 10)
after tag: 1 len: 3 ptr: 104423810 root: 104423800:64 attr: NO_SCAN
APPENDABLE


This is good information, thanks. I will get back to you with a druntime 
branch to try. Can I email you at this address? If not, email me at the 
address from my post to let me know your contact, no reason to work 
through building issues on the public forum :)


-Steve

Re: Garbage collector collects live objects

[Ruslan Mullakhmetov via Digitalmars-d-learn]

On Friday, 12 December 2014 at 15:50:26 UTC, Steven Schveighoffer 
wrote:


Can I email you at this address? If not, email me at the 
address from my post to let me know your contact, no reason to 
work through building issues on the public forum :)


-Steve


reach me at theambient [] me__com

Re: Garbage collector collects live objects

[Steven Schveighoffer via Digitalmars-d-learn]

On 12/10/14 7:52 AM, Ruslan Mullakhmetov wrote:

On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan Mullakhmetov wrote:

yes. that was the mistake. also after fixing bug in Blk Attributes
printing i got more reasonable attrs

for object blk: FINALIZE
for array of objects blk: NO_SCAN APPENDABLE

this is sound good except for NO_SCAN.

...
the other question why this happens... try to debug more.


I've done more dubugging.

what i've found:

initially array blk has only attrs APPENDABLE, but after some time this
blk is shrinked and reallocated (moved) and then NO_SCAN attr appears.


here the output of my extended logs:


before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr: APPENDABLE
after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr: NO_SCAN
APPENDABLE


this is produced by the following code

http://dpaste.dzfl.pl/0c6dc16270a1

so in a nutshell after appending to array via ~= operator blk attrs
changed from APPENDABLE to NO_SCAN APPENDABLE which cause the problem.

why and how this happens? can anybody explain it to me?


I have an idea of what is happening, I will do some testing. Thanks for 
debugging this so far, this is useful info.


This is *definitely* a bug, if the code you gave is what caused that 
output. Appending should not add the NO_SCAN tag.


-Steve

Re: Garbage collector collects live objects

[Steven Schveighoffer via Digitalmars-d-learn]

On 12/10/14 7:52 AM, Ruslan Mullakhmetov wrote:

On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan Mullakhmetov wrote:

yes. that was the mistake. also after fixing bug in Blk Attributes
printing i got more reasonable attrs

for object blk: FINALIZE
for array of objects blk: NO_SCAN APPENDABLE

this is sound good except for NO_SCAN.

...
the other question why this happens... try to debug more.


I've done more dubugging.

what i've found:

initially array blk has only attrs APPENDABLE, but after some time this
blk is shrinked and reallocated (moved) and then NO_SCAN attr appears.


here the output of my extended logs:


before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr: APPENDABLE
after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr: NO_SCAN
APPENDABLE



My analysis so far:

1. The before/after makes sense except for the attribute and the offset. 
A realloc into a 64-byte block should NOT cause an offset of 16 bytes. I 
have found why it's happening, which is a bug, but not one that should 
cause the problem of setting the noscan bit (will file an issue on that).
2. In the array append code, the block attributes are obtained via 
GC.query, which has this code for getting the attributes:


https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1792

Quoting from that function:

// reset the offset to the base pointer, otherwise the bits
// are the bits for the pointer, which may be garbage
offset = cast(size_t)(info.base - pool.baseAddr);
info.attr = getBits(pool, cast(size_t)(offset  pool.shiftBy));

Which should get the correct bits. I suspected there was an issue with 
getting the wrong bits, but this code looks correct.


3. The runtime caches the block info for thread local data for append 
speed. A potential issue is that the attributes are cached from a 
previous use for that block, but the GC (and the runtime itself) SHOULD 
clear that cache entry when that block is freed, avoiding this issue. A 
potential way to check this is to assert in a debug build of druntime 
that the cached block info always equals the actual block info. Are you 
able to build a debug version of druntime to test this? I can give you 
the changes you should make. This would explain the great difficulty in 
reproducing the issue.


4. If your code is multi-threaded, but using __gshared, it can make the 
cache incorrect. Are you doing this?


But the cache is really the only possible place I can see where the bits 
are set incorrectly, given that you just verified the bits are correct 
before the append.


Can you just list the version of the compiler you are using? I want to 
make sure this isn't an issue that has already been fixed.


-Steve

Re: Garbage collector collects live objects

[Ruslan Mullakhmetov via Digitalmars-d-learn]

On Wednesday, 10 December 2014 at 02:43:19 UTC, ketmar via 
Digitalmars-d-learn wrote:

On Tue, 09 Dec 2014 17:18:44 +
Ruslan Mullakhmetov via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:

but i still have no clue how to overcome GC =(
why do you want to fight with GC? most of the time GC is your 
friend.




see the topic: i got corruption when dereferencing object.

Re: Garbage collector collects live objects

[ketmar via Digitalmars-d-learn]

On Wed, 10 Dec 2014 08:32:12 +
Ruslan Mullakhmetov via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:

 On Wednesday, 10 December 2014 at 02:43:19 UTC, ketmar via 
 Digitalmars-d-learn wrote:
  On Tue, 09 Dec 2014 17:18:44 +
  Ruslan Mullakhmetov via Digitalmars-d-learn
  digitalmars-d-learn@puremagic.com wrote:
  but i still have no clue how to overcome GC =(
  why do you want to fight with GC? most of the time GC is your 
  friend.
 
 
 see the topic: i got corruption when dereferencing object.
that is easily fixable: just stop dereferencing it! ;-)


signature.asc
Description: PGP signature

Re: Garbage collector collects live objects

[Ruslan Mullakhmetov via Digitalmars-d-learn]

On Tuesday, 9 December 2014 at 21:38:57 UTC, Steven Schveighoffer 
wrote:

On 12/9/14 2:56 PM, Steven Schveighoffer wrote:

On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote:


array holds 11 64bit pointers but it's block size is only 128 
bytes  11

* 64 = 704 bytes. what's wrong with this arithmetics?


Hah, just realized what's wrong. It's not 64 *bytes* per 
pointer, it's 64 *bits*. So 8 bytes.


11 * 8 == 88.

Starting to sound more and more normal...

-Steve



yes. that was the mistake. also after fixing bug in Blk 
Attributes printing i got more reasonable attrs


for object blk: FINALIZE
for array of objects blk: NO_SCAN APPENDABLE

this is sound good except for NO_SCAN.


I did simple test file in which allocate array of Foo objects 
(http://dpaste.dzfl.pl/89ab00a897f6)


there i see blk attrs only APPENDABLE without NO_SCAN.

as far as i understand GC will not scan this array for references 
and those if the only reference to object is stored in this array 
will not see it, those assume this object as **not** referenced 
and collects it, am i right?


the other question why this happens... try to debug more.

Re: Garbage collector collects live objects

[Ruslan Mullakhmetov via Digitalmars-d-learn]

On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan 
Mullakhmetov wrote:
yes. that was the mistake. also after fixing bug in Blk 
Attributes printing i got more reasonable attrs


for object blk: FINALIZE
for array of objects blk: NO_SCAN APPENDABLE

this is sound good except for NO_SCAN.

...
the other question why this happens... try to debug more.


I've done more dubugging.

what i've found:

initially array blk has only attrs APPENDABLE, but after some 
time this blk is shrinked and reallocated (moved) and then 
NO_SCAN attr appears.



here the output of my extended logs:


before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr: 
APPENDABLE
after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr: 
NO_SCAN APPENDABLE



this is produced by the following code

http://dpaste.dzfl.pl/0c6dc16270a1

so in a nutshell after appending to array via ~= operator blk 
attrs changed from APPENDABLE to NO_SCAN APPENDABLE which cause 
the problem.


why and how this happens? can anybody explain it to me?