Re: Safe reference counting cannot be implemented as a library

[Martin Nowak via Digitalmars-d]

On Monday, 7 December 2015 at 07:10:42 UTC, Sönke Ludwig wrote:
I've very likely missed that part of the discussion - what were 
the reasons to not use "scope" for this?


Yeah good point, it should be possible to reuse scope as method 
attribute. I just used @noescape as a placeholder for the 
discussion.

Re: Safe reference counting cannot be implemented as a library

[Sönke Ludwig via Digitalmars-d]

Am 01.11.2015 um 21:47 schrieb Martin Nowak:

On 10/27/2015 12:41 PM, Andrei Alexandrescu wrote:
- I'm not a fan of adding yet another attribute but as inference support
   is currently limited it seems we'd need an explicit attribute for
   public APIs.


I've very likely missed that part of the discussion - what were the 
reasons to not use "scope" for this?

Re: Safe reference counting cannot be implemented as a library

[deadalnix via Digitalmars-d]

On Saturday, 5 December 2015 at 22:06:33 UTC, sclytrack wrote:
On Monday, 2 November 2015 at 09:56:14 UTC, Ola Fosheim Grøstad 
wrote:

On Sunday, 1 November 2015 at 22:04:51 UTC, deadalnix wrote:
On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak 
wrote:

On 11/01/2015 09:51 PM, Martin Nowak wrote:



http://lists.puremagic.com/cgi-bin/mailman/listinfo/dlang-study


The last post on the study group ended a while ago. Does that 
mean it is not going forward. It suggest to talk about 
lifetime. Is that with or without talking about

reference counting simultaneously?



It means that now that all the bikescheding is behind, people can 
work on the hard problems. And it takes time. Especially when you 
have goto int he language.

Re: Safe reference counting cannot be implemented as a library

[sclytrack via Digitalmars-d]

On Monday, 2 November 2015 at 09:56:14 UTC, Ola Fosheim Grøstad 
wrote:

On Sunday, 1 November 2015 at 22:04:51 UTC, deadalnix wrote:

On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

On 11/01/2015 09:51 PM, Martin Nowak wrote:



http://lists.puremagic.com/cgi-bin/mailman/listinfo/dlang-study


The last post on the study group ended a while ago. Does that 
mean it is not going forward. It suggest to talk about lifetime. 
Is that with or without talking about

reference counting simultaneously?

What approach will be taken?


1) Will for every assignment to the object a hidden temporary be 
created by the compiler to keep the object alive.


RCObject obj = new RCObject();
Item item = obj.items[x];

_temp1 = obj;   //The compiler inserts this automatically and 
is not seen.

obj=new RCObject();


2) Will you disable the item by some magical way? Item might 
still be valid.


RCObject obj = new RCObject();
...
Item item = obj.items[x];

obj=new RCObject();
//item is no longer accessible beyond this point.
writeln(item); //error


3) Somehow disable the RCObject for the duration of the borrowing.

with(Item item = RCObject.items[x])
{
  writeln(item)
}


4) Solved by eating pizza or watching a movie ...



Maybe the discussion should be a tree.

Re: Safe reference counting cannot be implemented as a library

[Ola Fosheim Grøstad via Digitalmars-d]

On Sunday, 1 November 2015 at 22:04:51 UTC, deadalnix wrote:

On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

On 11/01/2015 09:51 PM, Martin Nowak wrote:

Any hint/numbers showing that this is actually useful?


Also doesn't a good backend optimizer already fuse writes?


Yes but you have this myth flying around that it is necessary 
for good RC, because language like C++ do implicit sharing, so 
RC must be done atomically, so the optimizer can't optimize.


Yes, ARC does the atomic RC too, but ARC is bound by prior focus 
on manual RC without compiler support and should not be used as 
an example to be followed.


D could go for a more generic solution with specialized 
restricted integer types for resource tracking with different 
capabilites. Same amount of work, but much more versatile.


The library could then use templated RCPointers with the 
capabilites as parameters which recast the resource tracking 
integer to the desired type which injects the appropriate 
assumptions to the optimizer before/after each operation on the 
integer.


E.g. for the non-transfer-to-other-threads/fiber integer counter 
type:


1. the optimizer needs to know that below a specific 
block/stackframe the test "counter==0" always fails.


2. semantics that ensures that  contexts (fiber/thread) only are 
allowed to subtract a value they already have added. Easy way out 
is RAII.

Re: Safe reference counting cannot be implemented as a library

[deadalnix via Digitalmars-d]

On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

On 11/01/2015 09:51 PM, Martin Nowak wrote:

Any hint/numbers showing that this is actually useful?


Also doesn't a good backend optimizer already fuse writes?


Yes but you have this myth flying around that it is necessary for 
good RC, because language like C++ do implicit sharing, so RC 
must be done atomically, so the optimizer can't optimize.

Re: Safe reference counting cannot be implemented as a library

[rsw0x via Digitalmars-d]

On Sunday, 1 November 2015 at 22:36:46 UTC, Andrei Alexandrescu 
wrote:

On 11/01/2015 03:51 PM, Martin Nowak wrote:

On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:
Unrelated, and a foreshadowing of the discussion on the 
lifetime mailing
list: the compiler has ample opportunity to fuse incs/decs 
together, so

the signatures of these functions is:

void opInc(uint delta);
void opDec(uint delta);


Any hint/numbers showing that this is actually useful?


Would be great to collect some, and generally get rigorous 
about the whole approach.


Implementing such a cross statement optimization is quite some 
work. If
this occurs often enough (in particular for shared classes 
with atomic

ref counting) it might be worth the effort.


Most reference counting techniques revolve around reducing 
mutation of the reference count. See e.g. 
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf.


So we need to show that many refcount updates take it from 1 to 
larger than 1 and back. According to 
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf, many objects have a reference count of just one; the "eclipse" benchmark has 31.8% objects with a refcount greater than 1.



Andrei


That paper is assuming that you take Java(a language that does 
*not* have allocation patterns like D such as favoring data on 
the stack, tightly packed arrays of data, and immutability) rip 
out its GC, and replace it with a RC-based GC with no concept of 
unique ownership - no?

Re: Safe reference counting cannot be implemented as a library

[rsw0x via Digitalmars-d]

On Sunday, 1 November 2015 at 20:55:00 UTC, Martin Nowak wrote:

On 11/01/2015 09:51 PM, Martin Nowak wrote:

Any hint/numbers showing that this is actually useful?


Also doesn't a good backend optimizer already fuse writes?


AFAIK the fear of RC being too slow comes from C++'s shared_ptr's 
reference count being required to be synchronized by the C++ 
spec, which heavily limits what kinds of optimizations that can 
be done on it. Thread-local by default RC should, in theory, be 
able to be optimized much much more aggressively.

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 11/1/15 5:52 PM, rsw0x wrote:

On Sunday, 1 November 2015 at 22:36:46 UTC, Andrei Alexandrescu wrote:

On 11/01/2015 03:51 PM, Martin Nowak wrote:

On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:

Unrelated, and a foreshadowing of the discussion on the lifetime
mailing
list: the compiler has ample opportunity to fuse incs/decs together, so
the signatures of these functions is:

void opInc(uint delta);
void opDec(uint delta);


Any hint/numbers showing that this is actually useful?


Would be great to collect some, and generally get rigorous about the
whole approach.


Implementing such a cross statement optimization is quite some work. If
this occurs often enough (in particular for shared classes with atomic
ref counting) it might be worth the effort.


Most reference counting techniques revolve around reducing mutation of
the reference count. See e.g.
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf.

So we need to show that many refcount updates take it from 1 to larger
than 1 and back. According to
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf,
many objects have a reference count of just one; the "eclipse"
benchmark has 31.8% objects with a refcount greater than 1.


Andrei


That paper is assuming that you take Java(a language that does *not*
have allocation patterns like D such as favoring data on the stack,
tightly packed arrays of data, and immutability) rip out its GC, and
replace it with a RC-based GC with no concept of unique ownership - no?


The class objects we're focusing on for RC support are supposed to be 
used much like in Java. -- Andrei

Re: Safe reference counting cannot be implemented as a library

[Martin Nowak via Digitalmars-d]

On 10/27/2015 12:41 PM, Andrei Alexandrescu wrote:
> It follows that if we want safe reference counting, there must be
> language support for it. One possibility is to attach an attribute to
> the class definition:
> 
> @safe @rc class Widget {
>   ...
> }

Let's think about this more clearly before we bake a monolithic feature
for a single problem into the language.

A few thoughts:

- @rc and @noescape are orthogonal

  while @rc requires @noescape the latter could
  be useful for other purposes

- If the compiler knows a reference has a limited lifetime
  it could check for @noescape making most of RC implementable
  in a library.

  struct RC
  {
Object get() return; // lifetime of Object is bound to RC, compiler
could check any function called on Object for @noescape
  }

- I'm not a fan of adding yet another attribute but as inference support
  is currently limited it seems we'd need an explicit attribute for
  public APIs.

Re: Safe reference counting cannot be implemented as a library

[Martin Nowak via Digitalmars-d]

On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:
> Unrelated, and a foreshadowing of the discussion on the lifetime mailing
> list: the compiler has ample opportunity to fuse incs/decs together, so
> the signatures of these functions is:
> 
> void opInc(uint delta);
> void opDec(uint delta);

Any hint/numbers showing that this is actually useful?
Implementing such a cross statement optimization is quite some work. If
this occurs often enough (in particular for shared classes with atomic
ref counting) it might be worth the effort.

Re: Safe reference counting cannot be implemented as a library

[Martin Nowak via Digitalmars-d]

On 11/01/2015 09:51 PM, Martin Nowak wrote:
> Any hint/numbers showing that this is actually useful?

Also doesn't a good backend optimizer already fuse writes?

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 11/01/2015 03:54 PM, Martin Nowak wrote:

On 11/01/2015 09:51 PM, Martin Nowak wrote:

Any hint/numbers showing that this is actually useful?


Also doesn't a good backend optimizer already fuse writes?


My understanding is that no, that won't happen in most patterns that 
matter. -- Andrei

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 11/01/2015 03:51 PM, Martin Nowak wrote:

On 10/27/2015 01:27 PM, Andrei Alexandrescu wrote:

Unrelated, and a foreshadowing of the discussion on the lifetime mailing
list: the compiler has ample opportunity to fuse incs/decs together, so
the signatures of these functions is:

void opInc(uint delta);
void opDec(uint delta);


Any hint/numbers showing that this is actually useful?


Would be great to collect some, and generally get rigorous about the 
whole approach.



Implementing such a cross statement optimization is quite some work. If
this occurs often enough (in particular for shared classes with atomic
ref counting) it might be worth the effort.


Most reference counting techniques revolve around reducing mutation of 
the reference count. See e.g. 
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf.


So we need to show that many refcount updates take it from 1 to larger 
than 1 and back. According to 
https://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf, 
many objects have a reference count of just one; the "eclipse" benchmark 
has 31.8% objects with a refcount greater than 1.



Andrei

Re: Safe reference counting cannot be implemented as a library

[sclytrack via Digitalmars-d]

On Thursday, 29 October 2015 at 20:31:49 UTC, Zach the Mystic 
wrote:

On Tuesday, 27 October 2015 at 18:10:18 UTC, deadalnix wrote:
I've made the claim that we should implement reference 
counting as a library many time, so I think I should explicit 
my position. Indeed, RC require some level a compiler support 
to be safe. That being said, the support does not need to be 
specific to RC. On fact, my position is that the language 
should provide some basic mechanism on top of which safe RC 
can be implemented, as a library.


The problem at hand here is escape analysis. The compiler must 
be able to ensure that a reference doesn't escape the RC 
mechanism in an uncontrolled manner. I'd like to add such 
mechanism to the language rather than bake in reference 
counting, as it can be used to solve other problem we are 
facing today (@nogc exception for instance).


Here's a link to the reference safety system I proposed some 
months ago:


http://forum.dlang.org/post/offurllmuxjewizxe...@forum.dlang.org

I'm very far from having the expertise needed to know whether 
it would be worth its weight in practice, but it was better to 
write it out than to keep it bottled up in my head. I hope it 
will be of some use.



1) Assignment to RCObject
-


{
RCObject obj = new RCObject()
Item item1 = obj.items[x];

_temp1 = obj;
obj = new RCObject();


_temp2 = obj;
obj = new RCObject();
}



2) Entering a subscope
--

With this I mean entering a subscope that assigns to the RCObject.


{
RCObject obj2 = new RCObject()
RCObject obj = new RCObject()


_temp1 = obj;   //(2) Entering a subscope
{
_temp2 = obj;   //(1) Assignment to RCObject
obj = new RCObject();
}
}


3) Leaving a scope.
---

The Item is not reference counted.

{
Item item1;
{
RCObject obj = new RCObject();
//item1 = obj.items[x]; //(3) Leaving subscope
}
}



{
RCObject obj = new RCObject();
Item item1;

_temp1 = obj;   //(2) Entering subscope
{
_temp2 = obj;   //(1) Assignement to RCObject
obj = new RCObject();
//item1 = obj.items[x]; //(3) Leaving subscope

_temp3 = obj;   //(1) Assignement to RCObject
obj = new RCObject();
}
}




4) RCObject joins a scope
-


{
_temp1 = obj.field.rcobject;//(4) RCObject joins current 
scope.
Item item1 = obj.field.rcobject.a.items[0];


//_temp1;   //(2) Entering subscope
{
_temp3 = obj.field.rcobject;//(1) Assignment to RCObject
obj.field.rcobject = new RCObject();
}

_temp4 = obj.field.rcobject;//(4) RCObject joins current 
scope.
item1 = obj.field.rcobject.a.items[2];
}

Re: Safe reference counting cannot be implemented as a library

[Zach the Mystic via Digitalmars-d]

On Tuesday, 27 October 2015 at 18:10:18 UTC, deadalnix wrote:
I've made the claim that we should implement reference counting 
as a library many time, so I think I should explicit my 
position. Indeed, RC require some level a compiler support to 
be safe. That being said, the support does not need to be 
specific to RC. On fact, my position is that the language 
should provide some basic mechanism on top of which safe RC can 
be implemented, as a library.


The problem at hand here is escape analysis. The compiler must 
be able to ensure that a reference doesn't escape the RC 
mechanism in an uncontrolled manner. I'd like to add such 
mechanism to the language rather than bake in reference 
counting, as it can be used to solve other problem we are 
facing today (@nogc exception for instance).


Here's a link to the reference safety system I proposed some 
months ago:


http://forum.dlang.org/post/offurllmuxjewizxe...@forum.dlang.org

I'm very far from having the expertise needed to know whether it 
would be worth its weight in practice, but it was better to write 
it out than to keep it bottled up in my head. I hope it will be 
of some use.

Re: Safe reference counting cannot be implemented as a library

[David Nadlinger via Digitalmars-d]

On Wednesday, 28 October 2015 at 11:21:17 UTC, Manu wrote:
RC is okay-ish in C++11 (with rval references), although it 
could be
much better, for instance, the type mangling/wrapping induced 
by this

sort of library solution always leads to awkward situations, ie,
'this' pointer in a method is not an RC object anymore! Methods 
can't

give out pointers to themselves (ie, signaling events where it's
conventional to pass a 'sender' to the subscribers). Pretty 
massive

fail!


Did you look into doing something like 
std::enable_shared_from_this? I use it pretty routinely in 
networking code (boost.asio), and while it is not as pretty as it 
could be, it does the trick.


 – David

Re: Safe reference counting cannot be implemented as a library

[Manu via Digitalmars-d]

On 28 October 2015 at 11:13, Walter Bright via Digitalmars-d
 wrote:
> On 10/27/2015 11:10 AM, deadalnix wrote:
>>
>> I've made the claim that we should implement reference counting as a
>> library
>> many time, so I think I should explicit my position. Indeed, RC require
>> some
>> level a compiler support to be safe. That being said, the support does not
>> need
>> to be specific to RC. On fact, my position is that the language should
>> provide
>> some basic mechanism on top of which safe RC can be implemented, as a
>> library.
>
>
>
> It's not just safety. If the compiler knows that reference counting is going
> on, it can potentially elide a lot of the overhead. If it is faced with an
> arbitrary library solution, it only has a worm's eye view of it, and cannot
> do higher level optimizations.

I just want to drop in that I strongly feel both points here, they are
not at odds. I've been arguing for years now that D needs effective
escape analysis, this will allow all sorts of safe allocation and
lifetime patterns; and while it may enable some improved library
solutions to refcounting, I think the key advantage is actually
related to making better and safe use of stack allocation. I think
that is a much better focus when considering the need for
comprehensive escape analysis tools.
That has little to do with the language also benefiting from RC
primitives such that the compiler is able to do a quality job of
optimising ref-counting, which is a spectacularly prevalent pattern,
particularly so when used in conjunction with libraries such that the
inc/dec functions are opaque indirect calls into some foreign lib and
can't be optimised (this is the majority case in my experience). If
they can't be wrapped by a language primitive that it knows can
optimise this particular calling pattern, then the compiler has no
power to optimise such opaque calls at all.

As an anecdote, since I operate almost exclusively via practical
experience; my current project would heavily benefit from both, and
they would each contribute to a strong case for migration to D. These
2 issues alone represent, by far, the greatest trouble we face with
C++ currently.
RC is okay-ish in C++11 (with rval references), although it could be
much better, for instance, the type mangling/wrapping induced by this
sort of library solution always leads to awkward situations, ie,
'this' pointer in a method is not an RC object anymore! Methods can't
give out pointers to themselves (ie, signaling events where it's
conventional to pass a 'sender' to the subscribers). Pretty massive
fail!
But what we completely fail at is making good use of stack allocation;
requiring conservative fallback to heap allocations because we have no
proof mechanism for containing temporary ownership. We need expressive
escape analysis.

This is a very heavily object orientated codebase, rife with shared
pointers, with a strong focus on the external API and user
extensibility. Focus on the public API implies conservative allocation
habits; ie, RC is prevalent because we don't want to place complex
restrictions on users, and we must also be safe. If we has an
effective escape analysis mechanism, we would gain a lot of
opportunities to revert RC to stack allocations because we can
statically prove via the API that the user won't escape pointers.

The program consists of a typical hierarchical ownership structure, an
arbitrarily shared generalised resource pool, a highly interactive
scene graph with runtime datasets scaling to 10s of gigabytes, and a
ridiculously abstract API's (lots of painful C++ meta-magic). It is
also realtime. GC was considered and rejected on the premise that it
is realtime, and operates on truly gargantuan working datasets.
It's the most ambitious thing I've ever written, and I am dying inside
a little bit more every single day that I remain stuck with C++. I
want to start writing front-end plugin code in D as soon as possible,
which means, at very least, comprehensive RC interaction.

Re: Safe reference counting cannot be implemented as a library

[Manu via Digitalmars-d]

On 28 October 2015 at 21:29, David Nadlinger via Digitalmars-d
 wrote:
> On Wednesday, 28 October 2015 at 11:21:17 UTC, Manu wrote:
>>
>> RC is okay-ish in C++11 (with rval references), although it could be
>> much better, for instance, the type mangling/wrapping induced by this
>> sort of library solution always leads to awkward situations, ie,
>> 'this' pointer in a method is not an RC object anymore! Methods can't
>> give out pointers to themselves (ie, signaling events where it's
>> conventional to pass a 'sender' to the subscribers). Pretty massive
>> fail!
>
>
> Did you look into doing something like std::enable_shared_from_this? I use
> it pretty routinely in networking code (boost.asio), and while it is not as
> pretty as it could be, it does the trick.

I did. We ended up with custom solutions, tailored to take advantage
of our specific and fairly unconventional environment. Our RC is
invasive in most cases, using a duck-typed approach to the general
API.
I'm not really proud that we re-invented that wheel (among many,
because C++ offerings are generally insufficient), but it had to be
done.

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 10/28/2015 04:13 AM, Jacob Carlborg wrote:

On 2015-10-27 22:19, Andrei Alexandrescu wrote:


That doesn't seem to be the case at all. -- Andrei


I'm not a C++ or Rust expert. But I think that in Rust and with the new
C++ guide lines the idea is to use reference counting pointers only for
owning resources. If you want to pass the data to some of part of the
code, that does not need to own the resource, a raw pointer should be used.


Problem with that is in C++ it's just unsafe and in Rust it's requires 
too much work from the programmer. We don't think either style is best 
for D. -- Andrei

Re: Safe reference counting cannot be implemented as a library

[Paulo Pinto via Digitalmars-d]

On Wednesday, 28 October 2015 at 03:55:25 UTC, deadalnix wrote:
On Wednesday, 28 October 2015 at 01:13:16 UTC, Walter Bright 
wrote:


It's not just safety. If the compiler knows that reference 
counting is going on, it can potentially elide a lot of the 
overhead. If it is faced with an arbitrary library solution, 
it only has a worm's eye view of it, and cannot do higher 
level optimizations.


I don't think the compiler can do that much more, but before I 
address that point, let me mention that intrinsic can be added 
for inc and dec, which would be much more lightweight for the 
language at large.


Now as to why I think this wouldn't give that much. First, if 
exception can be thrown, then all bets are pretty much off, as 
inc and dec do not go by pairs anymore. So we are down to the 
no exception situation. In that case, pairs are fairly visible 
to the compiler and can be optimized away or combined, that is 
already the kind of things that optimizer are good at.


But if, so, how do you explain the C++ situation, where nothing 
is elided (or ObjC's) ?


Well, there is a major difference with these languages: sharing 
by default. It means that inc and dec must be (atomic and 
ordered) or synchronized, which means that, as far as the 
compiler is concerned, all bets are off and the optimizer can't 
do its job.


This doesn't really apply to D, so I don't expect this to be a 
problem. And even if there is: intrinsic can save the day to 
hint the optimizer, no need for heavyweight language addition.


Now, let's get back to the exception case, as it is IMO the 
most interesting one. What if one is willing to accept leakage 
on exception throwing. That would get the optimizer back into 
the game and remove a lot of "dark matter" as Andrei calls it, 
which have a real cost on term of icache pressure and exception 
unwinding cost (one now doesn't have to resume each frame to 
maintain refcount).


If I had to go about this, I'd rather see the introduction a 
scope(exit/success/failure) like mechanism for destructors 
rather than something ref counting specific.


Objective-C  does elide refcounting, there are a few WWDC ARC 
sessions where it is mentioned. Same applies to Swift.


However their exceptions work in a more RC friendly way.

Re: Safe reference counting cannot be implemented as a library

[Jacob Carlborg via Digitalmars-d]

On 2015-10-27 22:50, Andrei Alexandrescu wrote:


You can safely ignore the C++ part, the views are unsafe. I'd appreciate
if you backed up your claim on Rust. -- Andrei


Rust is unsafe as well, when you interface with unsafe code.

--
/Jacob Carlborg

Re: Safe reference counting cannot be implemented as a library

[Jacob Carlborg via Digitalmars-d]

On 2015-10-28 07:07, Paulo Pinto wrote:


However their exceptions work in a more RC friendly way.


Swift doesn't support exceptions. And in Objective-C exceptions are like 
Errors in D. They should not be caught and the program should terminate.


The error handling support that was added in Swift 2.0 is syntax sugar 
for the Objective-C pattern to use NSError out parameters for error 
handling.


--
/Jacob Carlborg

Re: Safe reference counting cannot be implemented as a library

[Jacob Carlborg via Digitalmars-d]

On 2015-10-27 22:19, Andrei Alexandrescu wrote:


That doesn't seem to be the case at all. -- Andrei


I'm not a C++ or Rust expert. But I think that in Rust and with the new 
C++ guide lines the idea is to use reference counting pointers only for 
owning resources. If you want to pass the data to some of part of the 
code, that does not need to own the resource, a raw pointer should be used.


--
/Jacob Carlborg

Re: Safe reference counting cannot be implemented as a library

[Walter Bright via Digitalmars-d]

On 10/28/2015 7:58 AM, Andrei Alexandrescu wrote:

On 10/28/2015 04:13 AM, Jacob Carlborg wrote:

On 2015-10-27 22:19, Andrei Alexandrescu wrote:


That doesn't seem to be the case at all. -- Andrei


I'm not a C++ or Rust expert. But I think that in Rust and with the new
C++ guide lines the idea is to use reference counting pointers only for
owning resources. If you want to pass the data to some of part of the
code, that does not need to own the resource, a raw pointer should be used.


Problem with that is in C++ it's just unsafe and in Rust it's requires too much
work from the programmer. We don't think either style is best for D. -- Andrei




Let's continue this in the mailing list.

Re: Safe reference counting cannot be implemented as a library

[Jacob Carlborg via Digitalmars-d]

On 2015-10-28 09:50, Paulo Pinto wrote:


Hence why I mentioned they are more RC friendly.

Swift, because it doesn't have them.

Objective-C, because termination is the only option so no need to worry
about preserving counters.

I was typing on the phone, so didn't want to provide the full explanation.


Fair enough :)

--
/Jacob Carlborg

Re: Safe reference counting cannot be implemented as a library

[Paulo Pinto via Digitalmars-d]

On Wednesday, 28 October 2015 at 08:07:40 UTC, Jacob Carlborg 
wrote:

On 2015-10-28 07:07, Paulo Pinto wrote:


However their exceptions work in a more RC friendly way.


Swift doesn't support exceptions. And in Objective-C exceptions 
are like Errors in D. They should not be caught and the program 
should terminate.


The error handling support that was added in Swift 2.0 is 
syntax sugar for the Objective-C pattern to use NSError out 
parameters for error handling.


Hence why I mentioned they are more RC friendly.

Swift, because it doesn't have them.

Objective-C, because termination is the only option so no need to 
worry about preserving counters.


I was typing on the phone, so didn't want to provide the full 
explanation.


--
Paulo

Re: Safe reference counting cannot be implemented as a library

[Ola Fosheim Grøstad via Digitalmars-d]

On Wednesday, 28 October 2015 at 06:07:12 UTC, Paulo Pinto wrote:
Objective-C  does elide refcounting, there are a few WWDC ARC 
sessions where it is mentioned. Same applies to Swift.


Indeed, John McCall from Apple has already described how ARC 
works in these forums (astonishingly nobody felt like thanking 
him for the input... :-/):


http://forum.dlang.org/post/hgmhgirfervrsvcgh...@forum.dlang.org

To what extent you can elide inc/dec depends on how you define 
and track ownership and whether you do whole program analysis, of 
course.

Re: Safe reference counting cannot be implemented as a library

[Rikki Cattermole via Digitalmars-d]

On 28/10/15 12:41 AM, Andrei Alexandrescu wrote:

(Title is borrowed from Hans Boehm's famous "Threads cannot be
implemented as a library",
http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf.)

We want to implement safe reference counting in D, and the sentiment
that it can be done in a library (with no or minimal changes to the
language) is often echoed in the newsgroup. This post explains that
reference counting can be implemented as a library _only_ if safety is
forgone. If safety is desired, reference counting must be built into
language semantics.

The crux of the matter is modular typechecking. Consider the following
example:

// module widget.d
@safe class Widget {
   void fun() {
 g_widget = this;
   }
}
static Widget g_widget;
// end of module widget.d

This is a perfect valid D class, and safe too. The typechecker assumes
infinite lifetime for all Widget objects, and allows escaping the
reference to this from foo() into the global.

Numerous similar examples can be constructed involving more elaborate
escaping patterns than this simple global assignment.

Now, once the typechecker OKs module widget.d, the summary that all
other typechecking "sees" is:

@safe class Widget {
   void fun();
}

A library reference counting wrapper a la RC!Widget needs to allow calls
to fun(). Once that happens, the global will hold an alias to the
respective Widget object indefinitely, which means as soon as the
RC!Widget object is released by the reference counting protocol,
g_widget will become a dangling reference.

It follows that if we want safe reference counting, there must be
language support for it. One possibility is to attach an attribute to
the class definition:

@safe @rc class Widget {
   ...
}

Then the compiler is able to enforce more stringent typechecking on
Widget (for example, address of fields are not allowed to escape) and
also insert the appropriate reference counting logic. This will be a
topic discussed on the lifetime list.


Andrei


Thought: we have RTInfo where we can place anything we want for a type.
Perhaps we could explore this avenue a bit?

A library solution, but perhaps in druntime instead of e.g. Phobos.
Add in compiler hooks and wala. Language support for it.

Other random thoughts, a storage attribute cast(@rc)myRef;
On a class (as you shown) @rc class Widget {}

I think there is no one solution here. But similar behavior with 
slightly different semantics.


Anyway, the only wrong solution involves:

RefCount!T func(IAllocator alloc=theAllocator) {
return RefCount!T(alloc.make!T, alloc);
}

Which I'm already doing.
And now I want to join that group... great.

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 10/27/2015 07:57 AM, Manu via Digitalmars-d wrote:

On 27 October 2015 at 21:41, Andrei Alexandrescu via Digitalmars-d
 wrote:


It follows that if we want safe reference counting, there must be language
support for it. One possibility is to attach an attribute to the class
definition:

@safe @rc class Widget {
   ...
}


An attribute? Is presence of opInc()/opDec() insufficient? Would the
attribute signal fabrication of some default opInc/opDec operators
applicable for general use?


You're right, opInc/opDec detection would suffice.

Unrelated, and a foreshadowing of the discussion on the lifetime mailing 
list: the compiler has ample opportunity to fuse incs/decs together, so 
the signatures of these functions is:


void opInc(uint delta);
void opDec(uint delta);

For example, consider:

class Widget {
  void fun(Widget, Widget);
}
...
auto w = new Widget;
w.fun(w, w);

In this case the compiler may insert opInc with a value larger than 1 
prior to entering the call.



Andrei

Re: Safe reference counting cannot be implemented as a library

[Manu via Digitalmars-d]

On 27 October 2015 at 21:41, Andrei Alexandrescu via Digitalmars-d
 wrote:
>
> It follows that if we want safe reference counting, there must be language
> support for it. One possibility is to attach an attribute to the class
> definition:
>
> @safe @rc class Widget {
>   ...
> }

An attribute? Is presence of opInc()/opDec() insufficient? Would the
attribute signal fabrication of some default opInc/opDec operators
applicable for general use?

Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

(Title is borrowed from Hans Boehm's famous "Threads cannot be 
implemented as a library", 
http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf.)


We want to implement safe reference counting in D, and the sentiment 
that it can be done in a library (with no or minimal changes to the 
language) is often echoed in the newsgroup. This post explains that 
reference counting can be implemented as a library _only_ if safety is 
forgone. If safety is desired, reference counting must be built into 
language semantics.


The crux of the matter is modular typechecking. Consider the following 
example:


// module widget.d
@safe class Widget {
  void fun() {
g_widget = this;
  }
}
static Widget g_widget;
// end of module widget.d

This is a perfect valid D class, and safe too. The typechecker assumes 
infinite lifetime for all Widget objects, and allows escaping the 
reference to this from foo() into the global.


Numerous similar examples can be constructed involving more elaborate 
escaping patterns than this simple global assignment.


Now, once the typechecker OKs module widget.d, the summary that all 
other typechecking "sees" is:


@safe class Widget {
  void fun();
}

A library reference counting wrapper a la RC!Widget needs to allow calls 
to fun(). Once that happens, the global will hold an alias to the 
respective Widget object indefinitely, which means as soon as the 
RC!Widget object is released by the reference counting protocol, 
g_widget will become a dangling reference.


It follows that if we want safe reference counting, there must be 
language support for it. One possibility is to attach an attribute to 
the class definition:


@safe @rc class Widget {
  ...
}

Then the compiler is able to enforce more stringent typechecking on 
Widget (for example, address of fields are not allowed to escape) and 
also insert the appropriate reference counting logic. This will be a 
topic discussed on the lifetime list.



Andrei

Re: Safe reference counting cannot be implemented as a library

[Manu via Digitalmars-d]

On 27 October 2015 at 22:27, Andrei Alexandrescu via Digitalmars-d
 wrote:
> On 10/27/2015 07:57 AM, Manu via Digitalmars-d wrote:
>>
>> On 27 October 2015 at 21:41, Andrei Alexandrescu via Digitalmars-d
>>  wrote:
>>>
>>>
>>> It follows that if we want safe reference counting, there must be
>>> language
>>> support for it. One possibility is to attach an attribute to the class
>>> definition:
>>>
>>> @safe @rc class Widget {
>>>...
>>> }
>>
>>
>> An attribute? Is presence of opInc()/opDec() insufficient? Would the
>> attribute signal fabrication of some default opInc/opDec operators
>> applicable for general use?
>
>
> You're right, opInc/opDec detection would suffice.
>
> Unrelated, and a foreshadowing of the discussion on the lifetime mailing
> list: the compiler has ample opportunity to fuse incs/decs together, so the
> signatures of these functions is:
>
> void opInc(uint delta);
> void opDec(uint delta);
>
> For example, consider:
>
> class Widget {
>   void fun(Widget, Widget);
> }
> ...
> auto w = new Widget;
> w.fun(w, w);
>
> In this case the compiler may insert opInc with a value larger than 1 prior
> to entering the call.

Awesome. Hadn't thought of that. Nice catch!
Perhaps add me to the list. I don't think I have anything to add that
I haven't said before, but I have very high interest in this topic.

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 10/27/15 11:59 AM, Jonathan M Davis wrote:

On Tuesday, 27 October 2015 at 12:27:29 UTC, Andrei Alexandrescu wrote:

Unrelated, and a foreshadowing of the discussion on the lifetime
mailing list:


What's the lifetime mailing list?


To be created. -- Andrei

Re: Safe reference counting cannot be implemented as a library

[Jonathan M Davis via Digitalmars-d]

On Tuesday, 27 October 2015 at 12:27:29 UTC, Andrei Alexandrescu 
wrote:
Unrelated, and a foreshadowing of the discussion on the 
lifetime mailing list:


What's the lifetime mailing list?

- Jonathan M Davis

Re: Safe reference counting cannot be implemented as a library

[deadalnix via Digitalmars-d]

I've made the claim that we should implement reference counting 
as a library many time, so I think I should explicit my position. 
Indeed, RC require some level a compiler support to be safe. That 
being said, the support does not need to be specific to RC. On 
fact, my position is that the language should provide some basic 
mechanism on top of which safe RC can be implemented, as a 
library.


The problem at hand here is escape analysis. The compiler must be 
able to ensure that a reference doesn't escape the RC mechanism 
in an uncontrolled manner. I'd like to add such mechanism to the 
language rather than bake in reference counting, as it can be 
used to solve other problem we are facing today (@nogc exception 
for instance).

Re: Safe reference counting cannot be implemented as a library

[Joseph Rushton Wakeling via Digitalmars-d]

On Tuesday, 27 October 2015 at 18:10:18 UTC, deadalnix wrote:
The problem at hand here is escape analysis. The compiler must 
be able to ensure that a reference doesn't escape the RC 
mechanism in an uncontrolled manner. I'd like to add such 
mechanism to the language rather than bake in reference 
counting, as it can be used to solve other problem we are 
facing today (@nogc exception for instance).


I would also have a definite interest in this for the direction 
I'm considering taking with an alternative std.random design.


Technically, I suspect my particular use-case is covered by 
DIP25, but a more comprehensive solution to escape analysis would 
definitely make it more secure.

Re: Safe reference counting cannot be implemented as a library

[Namespace via Digitalmars-d]

On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

 Why care about this?

 Even Rust doesn't try to solve this problem...because it isn't 
really a problem in practice.


 In c++/rust code you use value/unique types 99.9% of the time, 
and reference counting is only for shared/aliased objects.



 Reference counting == rare and unimportant


+1

Re: Safe reference counting cannot be implemented as a library

[Jonathan M Davis via Digitalmars-d]

On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

 Why care about this?

 Even Rust doesn't try to solve this problem...because it isn't 
really a problem in practice.


 In c++/rust code you use value/unique types 99.9% of the time, 
and reference counting is only for shared/aliased objects.



 Reference counting == rare and unimportant


Really? I've seen tons of C++ code that's written using smart 
pointers with objects living on the heap which then get passed 
around all over the place.


Sure, a lot of stuff in D should be structs on the stack, but 
there are plenty of cases where you need stuff on the heap, in 
which case, you either have to let the GC take care of it (which 
means no deterministic destruction), have something specific own 
it and destroy it when it's no longer needed, or reference count 
it so that it gets destroyed immediately after it's no longer 
needed. For cases where you don't care about deterministic 
destruction, using the GC is fine, but for those cases where 
deterministic destruction is required (e.g. because the object 
currently has ownership of an OS resource), the GC doesn't cut 
it, and ref-counting is very much what's needed.


If/when we introduce ref-counting into the language, I fully 
expect that there will be a lot of D programs written which 
specifically use it in order to avoid the GC. And while in many 
cases, that's going to be an unnecessary, in some cases, it'll be 
a lifesaver.


As it stands, we can add ref-counting via libraries just fine, 
but it requires giving up on @safe, which we could probably live 
with, but it would make @safe a lot less valuable in the long 
run. So, a solution that enables @safe ref-counting in D would 
certainly be desirable, and we definitely need a language 
improvement of some kind if we want to be able to ref-count 
something like exceptions (be it by building ref-counting into 
the language like Andrei is proposing or by adding other features 
which enable us to build it in the library like deadalnix is 
proposing).


- Jonathan M Davis

Re: Safe reference counting cannot be implemented as a library

[PuglyWUgly via Digitalmars-d]

 Why care about this?

 Even Rust doesn't try to solve this problem...because it isn't 
really a problem in practice.


 In c++/rust code you use value/unique types 99.9% of the time, 
and reference counting is only for shared/aliased objects.



 Reference counting == rare and unimportant

Re: Safe reference counting cannot be implemented as a library

[Ola Fosheim Grøstad via Digitalmars-d]

On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

 Why care about this?

 Even Rust doesn't try to solve this problem...because it isn't 
really a problem in practice.


 In c++/rust code you use value/unique types 99.9% of the time, 
and reference counting is only for shared/aliased objects.


I agree with your sentiment that local reference counting is 
seldom needed, but shared reference counting is needed for shared 
cached objects. So shared (atomic) reference counting have a 
common and important use case.


Not sure if shared ref counting is addressed here at all though.

Re: Safe reference counting cannot be implemented as a library

[rsw0x via Digitalmars-d]

On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis 
wrote:

On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

 Why care about this?

 Even Rust doesn't try to solve this problem...because it 
isn't really a problem in practice.


 In c++/rust code you use value/unique types 99.9% of the 
time, and reference counting is only for shared/aliased 
objects.



 Reference counting == rare and unimportant


Really? I've seen tons of C++ code that's written using smart 
pointers with objects living on the heap which then get passed 
around all over the place.


if they're using shared_ptr all over the place, they're doing it 
wrong.

shared_ptr is supposed to be a last resort

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 10/27/2015 05:26 PM, rsw0x wrote:

It has been a great success for Rust, I rarely ever see RC used anywhere
in Rust code thanks to borrowing. The new C++ core guidelines are also
heavily based on this cf. *_view types in GSL.


You can safely ignore the C++ part, the views are unsafe. I'd appreciate 
if you backed up your claim on Rust. -- Andrei

Re: Safe reference counting cannot be implemented as a library

[deadalnix via Digitalmars-d]

On Tuesday, 27 October 2015 at 20:45:34 UTC, rsw0x wrote:
if they're using shared_ptr all over the place, they're doing 
it wrong.

shared_ptr is supposed to be a last resort


According to Herb Sutter, it is a zero cost abstraction, not like 
these pesky GC that makes your program slow.

Re: Safe reference counting cannot be implemented as a library

[PuglyWUgly via Digitalmars-d]

Really? I've seen tons of C++ code that's written using smart 
pointers with objects living on the heap which then get passed 
around all over the place.


 Yes there is lots of old C++ code that does this, largely 
because unique_ptr wasn't implementable until C++11 added R value 
refs and move semantics.


Sure, a lot of stuff in D should be structs on the stack, but 
there are plenty of cases where you need stuff on the heap, in 
which case, you either have to let the GC take care of it 
(which means no deterministic destruction), have something 
specific own it and destroy it when it's no longer needed, or 
reference count it so that it gets destroyed immediately after 
it's no longer needed.


 on the heap == unique_ptr
 on the heap + multiple owners = shared_ptr

 If you work in a codebase that properly uses unique types, like 
Rust does by default, you soon notice how rare multiple owners 
actually is.

Re: Safe reference counting cannot be implemented as a library

[bitwise via Digitalmars-d]

On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis 
wrote:

On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

 [...]


Really? I've seen tons of C++ code that's written using smart 
pointers with objects living on the heap which then get passed 
around all over the place.


[...]


I can't believe you actually took the time to dignify this with a 
response...but on the other hand, I previously had no opinion 
about @safety...But if its a reason to include language level ref 
counting, then I guess I'm all for it :)


   Bit

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 10/27/2015 04:19 PM, PuglyWUgly wrote:

  Reference counting == rare and unimportant


That doesn't seem to be the case at all. -- Andrei

Re: Safe reference counting cannot be implemented as a library

[rsw0x via Digitalmars-d]

On Tuesday, 27 October 2015 at 21:50:15 UTC, Andrei Alexandrescu 
wrote:

On 10/27/2015 05:26 PM, rsw0x wrote:
It has been a great success for Rust, I rarely ever see RC 
used anywhere
in Rust code thanks to borrowing. The new C++ core guidelines 
are also

heavily based on this cf. *_view types in GSL.


You can safely ignore the C++ part, the views are unsafe. I'd 
appreciate if you backed up your claim on Rust. -- Andrei


rustc's source has about a 5:1 Box:Rc usage ratio after a quick 
grepping of the codebase(ignoring tests ofcourse.) This isn't 
counting all the cases where borrowed pointers were used instead 
of RC(which has about a 33.4:1 Borrowed:Rc usage after a quick 
grepping)

Re: Safe reference counting cannot be implemented as a library

[PuglyWUgly via Digitalmars-d]

You can safely ignore the C++ part, the views are unsafe. I'd 
appreciate if you backed up your claim on Rust. -- Andrei



 I did a rough check of the Rust compiler source(from a copy I 
downloaded a couple months ago).


I think the compiler is supposedly filled with old code, not sure 
if it is the best example of Rust.


 I filtered out the rc/arc.rs files, and doc/test folders

 Some of these hits were comments/commented out, but I'm too lazy 
to bother removing them


Anyway:

Searching 7524 files for "Rc::new(" (case sensitive)

122 matches across 49 files

Searching 7524 files for "Arc::new(" (case sensitive)

60 matches across 28 files

Re: Safe reference counting cannot be implemented as a library

[deadalnix via Digitalmars-d]

On Tuesday, 27 October 2015 at 21:50:15 UTC, Andrei Alexandrescu 
wrote:

On 10/27/2015 05:26 PM, rsw0x wrote:
It has been a great success for Rust, I rarely ever see RC 
used anywhere
in Rust code thanks to borrowing. The new C++ core guidelines 
are also

heavily based on this cf. *_view types in GSL.


You can safely ignore the C++ part, the views are unsafe. I'd 
appreciate if you backed up your claim on Rust. -- Andrei


That is ultimately irrelevant. Be it unique_ptr or shared_ptr, 
the same problem arise: this must not escape.

Re: Safe reference counting cannot be implemented as a library

[rsw0x via Digitalmars-d]

On Tuesday, 27 October 2015 at 21:20:58 UTC, Andrei Alexandrescu 
wrote:

On 10/27/2015 04:45 PM, rsw0x wrote:
On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis 
wrote:

On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

 Why care about this?

 Even Rust doesn't try to solve this problem...because it 
isn't

really a problem in practice.

 In c++/rust code you use value/unique types 99.9% of the 
time, and

reference counting is only for shared/aliased objects.


 Reference counting == rare and unimportant


Really? I've seen tons of C++ code that's written using smart 
pointers
with objects living on the heap which then get passed around 
all over

the place.


if they're using shared_ptr all over the place, they're doing 
it wrong.

shared_ptr is supposed to be a last resort


This is awesomely Kafkian. So we have no problem after all - 
just like Go with generics :o). -- Andrei


It has been a great success for Rust, I rarely ever see RC used 
anywhere in Rust code thanks to borrowing. The new C++ core 
guidelines are also heavily based on this cf. *_view types in GSL.


The problem D has is that RC isn't even implemented at library 
level let alone in a state where it's unsafe. IIRC Phobos RC 
doesn't even allow classes.

Re: Safe reference counting cannot be implemented as a library

[PuglyWUgly via Digitalmars-d]

 Well perhaps D prefers the Swift route, with everyone ref 
counted & some compiler help.


Nothing wrong with that I guess, I haven't been paying attention 
to what D is planning exactly.

Re: Safe reference counting cannot be implemented as a library

[Andrei Alexandrescu via Digitalmars-d]

On 10/27/2015 04:45 PM, rsw0x wrote:

On Tuesday, 27 October 2015 at 20:41:49 UTC, Jonathan M Davis wrote:

On Tuesday, 27 October 2015 at 20:19:42 UTC, PuglyWUgly wrote:

 Why care about this?

 Even Rust doesn't try to solve this problem...because it isn't
really a problem in practice.

 In c++/rust code you use value/unique types 99.9% of the time, and
reference counting is only for shared/aliased objects.


 Reference counting == rare and unimportant


Really? I've seen tons of C++ code that's written using smart pointers
with objects living on the heap which then get passed around all over
the place.


if they're using shared_ptr all over the place, they're doing it wrong.
shared_ptr is supposed to be a last resort


This is awesomely Kafkian. So we have no problem after all - just like 
Go with generics :o). -- Andrei

Re: Safe reference counting cannot be implemented as a library

[Jonathan M Davis via Digitalmars-d]

On Tuesday, 27 October 2015 at 21:50:15 UTC, Andrei Alexandrescu 
wrote:

On 10/27/2015 05:26 PM, rsw0x wrote:
It has been a great success for Rust, I rarely ever see RC 
used anywhere
in Rust code thanks to borrowing. The new C++ core guidelines 
are also

heavily based on this cf. *_view types in GSL.


You can safely ignore the C++ part, the views are unsafe. I'd 
appreciate if you backed up your claim on Rust. -- Andrei


Well, even if the claims for Rust and C++ were both 100% true, 
the fact remains that we need to do something about ref-counting 
in D, because the alternative is the GC, which is not necessarily 
acceptable. We already put lots of stuff on the stack and 
encourage avoiding allocations (so, in that respect, we're 
already doing what it's being suggested C++ and Rust do) - but 
some code simply needs to allocate to do what it does, and while 
the GC actually works fantastically for that in many cases, there 
are cases where it doesn't - particularly when you need 
deterministic destruction. So, even if if ref-counting in D ends 
up being very rare, and even if we don't care about @safety, we 
still need a way to support ref-counting for classes (preferably 
a standard way). And since we do care about @safety, it behooves 
us to at least investigate what the possible solutions are for 
having @safe ref-counting in D (and preferably implement one of 
them) rather than simply accepting that ref-counting can't be 
@safe.


Personally, I think that one of the most important aspects of 
this is finding a way to make it possible to reasonably use 
exceptions in @nogc code. Right now, @nogc is almost synonymous 
with nothrow, which is not at all good IMHO. I do not want us to 
start doing stuff like returning error codes just so that our 
code can be @nogc.


- Jonathan M Davis

Re: Safe reference counting cannot be implemented as a library

[Walter Bright via Digitalmars-d]

On 10/27/2015 11:10 AM, deadalnix wrote:

I've made the claim that we should implement reference counting as a library
many time, so I think I should explicit my position. Indeed, RC require some
level a compiler support to be safe. That being said, the support does not need
to be specific to RC. On fact, my position is that the language should provide
some basic mechanism on top of which safe RC can be implemented, as a library.



It's not just safety. If the compiler knows that reference counting is going on, 
it can potentially elide a lot of the overhead. If it is faced with an arbitrary 
library solution, it only has a worm's eye view of it, and cannot do higher 
level optimizations.

Re: Safe reference counting cannot be implemented as a library

[Walter Bright via Digitalmars-d]

On 10/27/2015 9:03 AM, Andrei Alexandrescu wrote:

On 10/27/15 11:59 AM, Jonathan M Davis wrote:

What's the lifetime mailing list?


To be created. -- Andrei


And you'll be in it for life, so be sure you want to join :-)

Re: Safe reference counting cannot be implemented as a library

[deadalnix via Digitalmars-d]

On Wednesday, 28 October 2015 at 01:13:16 UTC, Walter Bright 
wrote:


It's not just safety. If the compiler knows that reference 
counting is going on, it can potentially elide a lot of the 
overhead. If it is faced with an arbitrary library solution, it 
only has a worm's eye view of it, and cannot do higher level 
optimizations.


I don't think the compiler can do that much more, but before I 
address that point, let me mention that intrinsic can be added 
for inc and dec, which would be much more lightweight for the 
language at large.


Now as to why I think this wouldn't give that much. First, if 
exception can be thrown, then all bets are pretty much off, as 
inc and dec do not go by pairs anymore. So we are down to the no 
exception situation. In that case, pairs are fairly visible to 
the compiler and can be optimized away or combined, that is 
already the kind of things that optimizer are good at.


But if, so, how do you explain the C++ situation, where nothing 
is elided (or ObjC's) ?


Well, there is a major difference with these languages: sharing 
by default. It means that inc and dec must be (atomic and 
ordered) or synchronized, which means that, as far as the 
compiler is concerned, all bets are off and the optimizer can't 
do its job.


This doesn't really apply to D, so I don't expect this to be a 
problem. And even if there is: intrinsic can save the day to hint 
the optimizer, no need for heavyweight language addition.


Now, let's get back to the exception case, as it is IMO the most 
interesting one. What if one is willing to accept leakage on 
exception throwing. That would get the optimizer back into the 
game and remove a lot of "dark matter" as Andrei calls it, which 
have a real cost on term of icache pressure and exception 
unwinding cost (one now doesn't have to resume each frame to 
maintain refcount).


If I had to go about this, I'd rather see the introduction a 
scope(exit/success/failure) like mechanism for destructors rather 
than something ref counting specific.

Re: Safe reference counting cannot be implemented as a library

[rsw0x via Digitalmars-d]

On Wednesday, 28 October 2015 at 03:55:25 UTC, deadalnix wrote:
If I had to go about this, I'd rather see the introduction a 
scope(exit/success/failure) like mechanism for destructors 
rather than something ref counting specific.


can you expand upon this?

Re: Safe reference counting cannot be implemented as a library

[Sebastiaan Koppe via Digitalmars-d]

On Tuesday, 27 October 2015 at 11:41:52 UTC, Andrei Alexandrescu 
wrote:
The crux of the matter is modular typechecking. Consider the 
following example:


// module widget.d
@safe class Widget {
  void fun() {
g_widget = this;
  }
}
static Widget g_widget;
// end of module widget.d

Now, once the typechecker OKs module widget.d, the summary that 
all other typechecking "sees" is:


@safe class Widget {
  void fun();
}


Isn't it a shame that that kind of information gets tossed aside? 
Seems to be very valuable and the loss of it the cause of several 
issues.

Re: Safe reference counting cannot be implemented as a library

[Jonathan M Davis via Digitalmars-d]

On Wednesday, 28 October 2015 at 02:08:18 UTC, Sebastiaan Koppe 
wrote:
On Tuesday, 27 October 2015 at 11:41:52 UTC, Andrei 
Alexandrescu wrote:
The crux of the matter is modular typechecking. Consider the 
following example:


// module widget.d
@safe class Widget {
  void fun() {
g_widget = this;
  }
}
static Widget g_widget;
// end of module widget.d

Now, once the typechecker OKs module widget.d, the summary 
that all other typechecking "sees" is:


@safe class Widget {
  void fun();
}


Isn't it a shame that that kind of information gets tossed 
aside? Seems to be very valuable and the loss of it the cause 
of several issues.


Perhaps, but that assignment is perfectly @safe. What isn't @safe 
is when you then free the Widget later because a ref-count hit 0. 
And that code already has to be @system or @trusted given what 
it's doing. So, really, the problem is that some @safe operations 
can lead to problems when @system code does stuff that assumes 
that those operations did not occur. So, unfortunately, whether 
that code can be @trusted depends on what some @safe code did 
rather than @system code like you'd normally expect. But the 
compiler simply doesn't understand enough about why something is 
@system to be able to figure out where @safe code could foul it 
up even if it has all of the source code to look at.


- Jonathan M Davis