Re: DIP60: @nogc attribute
Here's another thing that should be allowed that doesn't depend
on optimizations:
Any code path in a @nogc function that is guaranteed to abort the
program should be exempt from @nogc enforcement. This includes
assert(0) and throwing an Error.
Take std.exception.assumeWontThrow() as an example:
T assumeWontThrow(T)(lazy T expr,
string msg = null,
string file = __FILE__,
size_t line = __LINE__) nothrow
{
try
{
return expr;
}
catch(Exception e)
{
immutable tail = msg.empty ? . : : ~ msg;
throw new AssertError(assumeWontThrow failed: Expression
did throw ~
tail, file, line);
}
}
Currently, this cannot be @nogc, because it uses `new` and `~`.
However, this only happens in preparation to throwing the
AssertError, which in turn causes the program to abort. I guess
in this situation, it's ok to allocate on the GC heap.
With my proposed rule, assumeWontThrow can be deduced to be @nogc
iff expr is @nogc. This allows more functions to be @nogc.
Re: DIP60: @nogc attribute
On 2014-04-26 16:43, Daniel Murphy wrote: At least these days it only happens when Walter and Andrei agree instead of just Walter merging whatever he feels like. Yeah, but it's still a problem when the rest of the community disagrees. -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 4/25/2014 7:28 AM, bearophile wrote: Dicebot: It is unacceptable to have code that fails with one compiler and works with the other despite the shared frontend version. Such enhanced @nogc attributes must be placed into compiler-specific attribute space and not as a core language feature. This problem was underlined during this thread far before the merging of the @nogc implementation. Why have Walter Andrei ignored the problem? What's the point of creating a DIP if you ignore the problems found in its discussion? What's the point of 338 comment posts if Walter goes on anyway with the original idea? There are some problems in the D development method that must be addressed. The @nogc logic is entirely contained in the front end, and is not affected by back end logic.
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 15:29:38 UTC, Steven Schveighoffer wrote: Well, @nogc is not released yet. Please tell me we don't have to avoid breaking code based on git HEAD... We've already done that before, with UDA's. So, you never know. -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 2014-04-25 16:28, bearophile wrote: This problem was underlined during this thread far before the merging of the @nogc implementation. Why have Walter Andrei ignored the problem? What's the point of creating a DIP if you ignore the problems found in its discussion? What's the point of 338 comment posts if Walter goes on anyway with the original idea? There are some problems in the D development method that must be addressed. That's a problem. The problem is if someone has an idea/code it wants to be merged, it's enough to convince one developer with push right to get it merged. -- /Jacob Carlborg
Re: DIP60: @nogc attribute
Walter Bright:
The @nogc logic is entirely contained in the front end, and is
not affected by back end logic.
Thank you for your answer and sorry for me being sometimes too
much nervous.
So the problem I was alarmed about doesn't exists.
Some time ago I have filed this ER:
https://issues.dlang.org/show_bug.cgi?id=12642
That shows this rejected code that I thought could be accepted:
__gshared int[1] data1;
int[1] bar() @nogc {
int x;
return [x];
}
void main() @nogc {
int x;
data1 = [x];
int[1] data2;
data2 = [x];
}
So that's an example of what you are talking about. DMD is
already performing some stack allocation of array literals that
the @nogc is not seeing and rejecting.
Kenji Hara has commented:
If you remove @nogc annotation, all array literals will be
allocated on stack. So this is pure front-end issue,
and may be fixed easily.
So the ER 12642 should be a wontfix, or a front-end rule should
be added to be added so all D compilers allocate those cases on
the stack.
If I am not missing some more point, what is the best solution?
Bye,
bearophile
Re: DIP60: @nogc attribute
Jacob Carlborg wrote in message news:ljfvec$126l$1...@digitalmars.com... That's a problem. The problem is if someone has an idea/code it wants to be merged, it's enough to convince one developer with push right to get it merged. At least these days it only happens when Walter and Andrei agree instead of just Walter merging whatever he feels like.
Re: DIP60: @nogc attribute
If I am not missing some more point, what is the best solution? Before this question gets lost, I'd like to receive some kind of answer. Thank you, bearophile
Re: DIP60: @nogc attribute
On 04/27/2014 01:32 AM, bearophile wrote: If I am not missing some more point, what is the best solution? Before this question gets lost, I'd like to receive some kind of answer. Thank you, bearophile The front end already distinguishes dynamic and static array literals (in a limited form), this distinction should simply carry through to code generation and static array literals should be allowed in @nogc code.
Re: DIP60: @nogc attribute
Walter Bright: Pointing out these issues is exactly what @nogc is designed to do. Using @nogc is like putting your code under a newly invented microscope, it allows to see things that I missed before :-) Bye, bearophile
Re: DIP60: @nogc attribute
On Thursday, 24 April 2014 at 13:35:39 UTC, bearophile wrote: immutable int x = 3; auto result = data[].map!(y = y * x); } test.d(1,6): Error: function D main @nogc function allocates a closure with the GC Such kind of code is common, so a good amount of range-based code can't be @nogc. Why can't this be on the stack if the referenced local function (lambda) does not outlive the stack frame?
Re: DIP60: @nogc attribute
On Fri, 25 Apr 2014 07:28:27 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Thursday, 24 April 2014 at 13:35:39 UTC, bearophile wrote: immutable int x = 3; auto result = data[].map!(y = y * x); } test.d(1,6): Error: function D main @nogc function allocates a closure with the GC Such kind of code is common, so a good amount of range-based code can't be @nogc. Why can't this be on the stack if the referenced local function (lambda) does not outlive the stack frame? It could. I don't think the compiler is smart enough, as it would need to verify result doesn't go anywhere (flow analysis). I wonder if LDC/GDC could do it. One interesting thing about this is that the compiler implementation may make some @nogc code valid on some compilers, and invalid on others, even though the resulting execution is the same. -Steve
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 12:07:00 UTC, Steven Schveighoffer wrote: One interesting thing about this is that the compiler implementation may make some @nogc code valid on some compilers, and invalid on others, even though the resulting execution is the same. I don't think this is a desirable behavior. @nogc should be decided in the frontend, before closure allocation optimizations take place. David
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 12:07:00 UTC, Steven Schveighoffer wrote: It could. I don't think the compiler is smart enough, as it would need to verify result doesn't go anywhere (flow analysis). In that case I'd like to see recursive inlining, if it makes stack allocations more probable.
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 12:21:40 UTC, David Nadlinger wrote: On Friday, 25 April 2014 at 12:07:00 UTC, Steven Schveighoffer wrote: One interesting thing about this is that the compiler implementation may make some @nogc code valid on some compilers, and invalid on others, even though the resulting execution is the same. I don't think this is a desirable behavior. @nogc should be decided in the frontend, before closure allocation optimizations take place. Yes, but the language specification should guarantee that no heap allocation takes place at least for some simple cases. `scope` comes to mind... This can apply to other normally allocating operations, too, like `new` and array concatenation/appending.
Re: DIP60: @nogc attribute
On Fri, 25 Apr 2014 08:21:38 -0400, David Nadlinger c...@klickverbot.at wrote: On Friday, 25 April 2014 at 12:07:00 UTC, Steven Schveighoffer wrote: One interesting thing about this is that the compiler implementation may make some @nogc code valid on some compilers, and invalid on others, even though the resulting execution is the same. I don't think this is a desirable behavior. @nogc should be decided in the frontend, before closure allocation optimizations take place. I don't know that it's desirable to have @nogc reject code even though an allocation does not occur. I agree the situation is not ideal, but @nogc is a practical optimization. I can think of other cases where the GC may be optimized out. To reject such code in @nogc would make it much less attractive. -Steve
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 12:59:55 UTC, Steven Schveighoffer wrote: On Fri, 25 Apr 2014 08:21:38 -0400, David Nadlinger c...@klickverbot.at wrote: On Friday, 25 April 2014 at 12:07:00 UTC, Steven Schveighoffer wrote: One interesting thing about this is that the compiler implementation may make some @nogc code valid on some compilers, and invalid on others, even though the resulting execution is the same. I don't think this is a desirable behavior. @nogc should be decided in the frontend, before closure allocation optimizations take place. I don't know that it's desirable to have @nogc reject code even though an allocation does not occur. I agree the situation is not ideal, but @nogc is a practical optimization. I can think of other cases where the GC may be optimized out. To reject such code in @nogc would make it much less attractive. -Steve It is unacceptable to have code that fails with one compiler and works with the other despite the shared frontend version. Such enhanced @nogc attributes must be placed into compiler-specific attribute space and not as a core language feature.
Re: DIP60: @nogc attribute
On Fri, 25 Apr 2014 09:20:08 -0400, Dicebot pub...@dicebot.lv wrote: On Friday, 25 April 2014 at 12:59:55 UTC, Steven Schveighoffer wrote: I don't know that it's desirable to have @nogc reject code even though an allocation does not occur. I agree the situation is not ideal, but @nogc is a practical optimization. I can think of other cases where the GC may be optimized out. To reject such code in @nogc would make it much less attractive. It is unacceptable to have code that fails with one compiler and works with the other despite the shared frontend version. Such enhanced @nogc attributes must be placed into compiler-specific attribute space and not as a core language feature. Like I said, this may be the ideologically correct position, but please explain to the poor user that even though the compiler does not invoke the GC in his function, it still cannot be @nogc. I think in this case, @nogc is not a good name. But what really is the difference between a function that is marked as @nogc that compiles on compiler X but not compiler Y, and some custom attribute that compiles on X but not Y? Consider that the fact that compiler X could have compiled a function that compiler Y is linking to, may actually be @nogc, because compiler X is better at avoiding GC calls. Wouldn't it make sense to be able to mark it @nogc and still use it from compiler Y? -Steve
Re: DIP60: @nogc attribute
Dicebot: It is unacceptable to have code that fails with one compiler and works with the other despite the shared frontend version. Such enhanced @nogc attributes must be placed into compiler-specific attribute space and not as a core language feature. This problem was underlined during this thread far before the merging of the @nogc implementation. Why have Walter Andrei ignored the problem? What's the point of creating a DIP if you ignore the problems found in its discussion? What's the point of 338 comment posts if Walter goes on anyway with the original idea? There are some problems in the D development method that must be addressed. Bye, bearophile
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 14:01:07 UTC, Steven Schveighoffer wrote: It is unacceptable to have code that fails with one compiler and works with the other despite the shared frontend version. Such enhanced @nogc attributes must be placed into compiler-specific attribute space and not as a core language feature. Like I said, this may be the ideologically correct position, but please explain to the poor user that even though the compiler does not invoke the GC in his function, it still cannot be @nogc. I think in this case, @nogc is not a good name. Which is the very reason why I was so insisting of defining exact set of cases when optimisation is to be guaranteed in spec (before releasing @nogc). Unfortunately, with no success. But what really is the difference between a function that is marked as @nogc that compiles on compiler X but not compiler Y, and some custom attribute that compiles on X but not Y? There are no user-defined attributes that can possibly fail on only some compiler. And compiler-specific attributes are part of that compiler documentation and never part of language spec. This is the difference.
Re: DIP60: @nogc attribute
On Fri, 25 Apr 2014 11:12:54 -0400, Dicebot pub...@dicebot.lv wrote: On Friday, 25 April 2014 at 14:01:07 UTC, Steven Schveighoffer wrote: It is unacceptable to have code that fails with one compiler and works with the other despite the shared frontend version. Such enhanced @nogc attributes must be placed into compiler-specific attribute space and not as a core language feature. Like I said, this may be the ideologically correct position, but please explain to the poor user that even though the compiler does not invoke the GC in his function, it still cannot be @nogc. I think in this case, @nogc is not a good name. Which is the very reason why I was so insisting of defining exact set of cases when optimisation is to be guaranteed in spec (before releasing @nogc). Unfortunately, with no success. Well, @nogc is not released yet. Please tell me we don't have to avoid breaking code based on git HEAD... But what really is the difference between a function that is marked as @nogc that compiles on compiler X but not compiler Y, and some custom attribute that compiles on X but not Y? There are no user-defined attributes that can possibly fail on only some compiler. And compiler-specific attributes are part of that compiler documentation and never part of language spec. This is the difference. But such a situation would not violate a spec that says @nogc means there are no hidden GC calls. And the end result is identical -- I must compile function foo on compiler X only. I agree there are likely no precedents for this. Another option would be to put such a compiler-specific attribute around the code in question rather than a different attribute than @nogc on the function itself. I think there's really no avoiding that this will happen some way or another. -Steve
Re: DIP60: @nogc attribute
On Fri, 25 Apr 2014 11:29:37 -0400, Steven Schveighoffer schvei...@yahoo.com wrote: On Fri, 25 Apr 2014 11:12:54 -0400, Dicebot pub...@dicebot.lv wrote: On Friday, 25 April 2014 at 14:01:07 UTC, Steven Schveighoffer wrote: But what really is the difference between a function that is marked as @nogc that compiles on compiler X but not compiler Y, and some custom attribute that compiles on X but not Y? There are no user-defined attributes that can possibly fail on only some compiler. And compiler-specific attributes are part of that compiler documentation and never part of language spec. This is the difference. But such a situation would not violate a spec that says @nogc means there are no hidden GC calls. And the end result is identical -- I must compile function foo on compiler X only. You know what, in fact, @nogc may need to be re-branded as compiler-specific. -Steve
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 15:29:38 UTC, Steven Schveighoffer wrote: On Fri, 25 Apr 2014 11:12:54 -0400, Dicebot pub...@dicebot.lv Which is the very reason why I was so insisting of defining exact set of cases when optimisation is to be guaranteed in spec (before releasing @nogc). Unfortunately, with no success. Well, @nogc is not released yet. Please tell me we don't have to avoid breaking code based on git HEAD... It has become a blocker for next release though. It has been repeated numerous times that such features need to be developed in own feature branches until design is considered at least somewhat solid =/ But what really is the difference between a function that is marked as @nogc that compiles on compiler X but not compiler Y, and some custom attribute that compiles on X but not Y? There are no user-defined attributes that can possibly fail on only some compiler. And compiler-specific attributes are part of that compiler documentation and never part of language spec. This is the difference. But such a situation would not violate a spec that says @nogc means there are no hidden GC calls. And the end result is identical -- I must compile function foo on compiler X only. It is invalid and useless spec on its own. It would have been a valid spec if it also had a chapter with definitive list of all cases when hidden GC calls can happen and when are guaranteed to be optimized away. Otherwise such spec is as useful as one that says Maybe your code will compile. I agree there are likely no precedents for this. Another option would be to put such a compiler-specific attribute around the code in question rather than a different attribute than @nogc on the function itself. I think there's really no avoiding that this will happen some way or another. I think there should be both. Saying that you need to marry specific compiler forever once you want to use @nogc is pretty much same as saying don't use @nogc.
Re: DIP60: @nogc attribute
On Friday, 25 April 2014 at 15:32:40 UTC, Steven Schveighoffer wrote: You know what, in fact, @nogc may need to be re-branded as compiler-specific. You should have a compiler switch that let's you get compiler-optimal non-portable @nogc.
Re: DIP60: @nogc attribute
On Fri, 25 Apr 2014 20:50:46 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Friday, 25 April 2014 at 15:32:40 UTC, Steven Schveighoffer wrote: You know what, in fact, @nogc may need to be re-branded as compiler-specific. You should have a compiler switch that let's you get compiler-optimal non-portable @nogc. I feel like I'm being a nag, but it sure seems to me like having something like that is no different than custom behavior of @nogc. Basically, I have a file blah.d, which can only be compiled with X, even if it's only with the option X -extraNogcFunctionality In other words, I have this function. It can avoid GC allocations if the compiler can do extra steps to prove it. But not all compilers go to these lengths. So if I only care about compiling with savvy enough compilers, why do I need to use some special compiler-specific escape? I think the attribute is fine being defined as if you can't do this without calling the GC, refuse to compile, and the compiler may or may not compile it. In any case, I don't need another explanation, I don't think it will ever make sense to me. -Steve
Re: DIP60: @nogc attribute
On Saturday, 26 April 2014 at 04:49:07 UTC, Steven Schveighoffer wrote: In any case, I don't need another explanation, I don't think it will ever make sense to me. It makes sense because there are two different use cases: 1. Library authors who need a more conservative interpretation of @nogc. 2. Binary release productions who only want to be certain that the GC is not called where it could lead to a crash. It would be annoying to have to rewrite code when the compiler actually knows that it does not touch the GC. So the latter use cases need the less conservative approach.
Re: DIP60: @nogc attribute
On 23/04/14 19:12, Walter Bright wrote: Too many double negatives for me to be sure what you're saying. But it is clear to me that with Michel's experience with ARC in iOS combined with Manu's enthusiasm for it suggests that they are the right team to come up with a workable proposal, where mine failed. Sorry, now that I read it out loud it is confusing. Here's another try: You're proposal wasn't compatible with ARC in Objective-C. I'm not sure if I remember correctly. -- /Jacob Carlborg
Re: DIP60: @nogc attribute
Walter Bright:
http://wiki.dlang.org/DIP60
Start on implementation:
https://github.com/D-Programming-Language/dmd/pull/3455
Currently this code doesn't compile because the lambda allocates
the closure on the heap:
void main() @nogc {
import std.algorithm: map;
int[3] data = [1, 2, 3];
immutable int x = 3;
auto result = data[].map!(y = y * x);
}
test.d(1,6): Error: function D main @nogc function allocates a
closure with the GC
Such kind of code is common, so a good amount of range-based code
can't be @nogc.
---
In the meantime the good Kenji has created a patch for missing
semantics:
https://github.com/D-Programming-Language/dmd/pull/3493
Bye,
bearophile
Re: DIP60: @nogc attribute
On 4/24/2014 6:35 AM, bearophile wrote:
Currently this code doesn't compile because the lambda allocates the closure on
the heap:
Pointing out these issues is exactly what @nogc is designed to do.
void main() @nogc {
import std.algorithm: map;
int[3] data = [1, 2, 3];
immutable int x = 3;
auto result = data[].map!(y = y * x);
}
test.d(1,6): Error: function D main @nogc function allocates a closure with the
GC
Such kind of code is common,
True.
so a good amount of range-based code can't be @nogc.
Can't is a bit strong of a word. Needing a workaround that is perhaps a bit
ugly is more accurate. For your example,
enum int x = 3;
will solve the issue.
Re: DIP60: @nogc attribute
Walter Bright: Pointing out these issues is exactly what @nogc is designed to do. Right. Can't is a bit strong of a word. Needing a workaround that is perhaps a bit ugly is more accurate. For your example, enum int x = 3; will solve the issue. In most cases that x is a run-time value, as in my example. Bye, bearophile
Re: DIP60: @nogc attribute
On 4/24/2014 11:49 AM, bearophile wrote: Walter Bright: Can't is a bit strong of a word. Needing a workaround that is perhaps a bit ugly is more accurate. For your example, enum int x = 3; will solve the issue. In most cases that x is a run-time value, as in my example. You can make it a static and it'll work. Ugly, but it'll work.
Re: DIP60: @nogc attribute
On 23/04/14 06:33, Walter Bright wrote: I repeatedly said that it is not memory safe because you must employ escapes from it to get performance. Apparently you need that for the GC as well, that's why this thread was started to begin with. -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 22/04/14 20:48, Steven Schveighoffer wrote: I mean not like I can't because I don't want to or don't have time, but can't as in I lack the skill set :) It's interesting to debate, and I get the concepts, but I am not a CPU/cache guy, and these things are really important to get right for performance, since ref counting would be used frequently. That's the worst kind of excuses :) I don't remember the last time I started working on a project and know what I was doing/had the right skill set. I mean, that's how you learn. -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 22 April 2014 05:03, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: On 4/21/2014 10:57 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 13:28:24 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/21/2014 5:00 AM, Steven Schveighoffer wrote: Total replacement of GC with ARC in D will: This is the wrong straw-man, I'm not advocating for this at all. Many are when they advocate ARC for D. Does that preclude you from accepting any kind of ARC for D? No. My objection is to pervasive ARC, i.e. all gc is replaced with ARC, and it all magically works. It's not magic, it's careful engineering, and considering each problem case one by one as they arise until it's good. 5. Numerous posters here have posited that the overhead of ARC can be eliminated with a sufficiently smart compiler (which does not exist). You continue to speak in extremes. People are saying that the compiler can eliminate most of the needless ARC increments and decrements, not all of them. Manu, for example, suggests it is good enough to make the overhead insignificant. I'm skeptical. I didn't quite say that, but let me justify that claim if you want to put it in those words. RC fiddling in low-frequency code is insignificant. High-frequency code doesn't typically allocate, and is also likely to implement a context specific solution anyway if it is truly performance sensitive. In the event of code where RC fiddling is found to make a significant impact on performance, there are various tools available to address this directly. There's a middle-ground that might suffer compared to GC; moderate-frequency, where code is sloppily written doing whatever it likes without any real care, and run lots of iterations. But that's not usually an example of performance sensitive code, it's just crappy code run many times, and again, they have the tools to improve it easily if they care enough to do so. I also believe programmers will learn the performance characteristics of ARC very quickly, and work with it effectively. The core of my argument is that it's _possible_ to work with ARC, it's not possible to work with GC if it is fundamentally incompatible with your application. Compilers that do this do exist. I can't reconcile agreeing that ARC isn't good enough to be pervasive with compiler technology eliminates unnecessary ARC overhead. The most important elimination is objects being passed down a call-tree via args. That can certainly eliminate properly. High-frequency code always exists nearer to the leaves. D has pure (which is definitely well employed), and 'shared' is an explicit attribute, which allows the compiler to make way more assumptions than O-C. The ARC optimisations are predictable and reliable. The suggestion that acceptable code performance relying on specific optimisation is a concept you can't reconcile is a bit strange. Programmers rely on optimisation all the time for acceptable performance. This is no different.
Re: DIP60: @nogc attribute
Manu, you obviously believe in ARC. I've made an attempt to do ARC, detailed in the other thread here. I failed. http://forum.dlang.org/thread/l34lei$255v$1...@digitalmars.com Michel Fortin also wants to bring iOS ARC to D. I suggest you get together with Michel and work out a detailed design, and propose it.
Re: DIP60: @nogc attribute
On 23 April 2014 04:28, Steven Schveighoffer via Digitalmars-d digitalmars-d@puremagic.com wrote: On Tue, 22 Apr 2014 14:12:17 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/22/2014 6:18 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 19:02:53 -0400, Walter Bright newshou...@digitalmars.com wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. So? If you see no value in static guarantees of memory safety, then what can I say? Seriously, the straw man arguments have to stop. There is plenty of valuable D code that is not guaranteed memory safe. For example, druntime. ARC does not equal guaranteed memory safety. So NO, it cannot replace the GC for D @safe code. That doesn't make it useless. Why not? Assuming that direct access to the refcount is not @safe, why would ARC be unsafe? What makes it less safe than the GC?
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 19:42:20 UTC, Michel Fortin wrote: Objective-C isn't memory safe because it lets you play with raw pointers too. If you limit yourself to ARC-managed pointers (and avoid undefined behaviours inherited from C) everything is perfectly memory safe. I'm not convinced that it is safe in multi-threaded mode. How does ARC deal with parallell reads and writes from two different threads? IIRC the most common implementations deals with read/read and write/write, but read/write is too costly?
Re: DIP60: @nogc attribute
On 23/04/14 10:31, Walter Bright wrote: Manu, you obviously believe in ARC. I've made an attempt to do ARC, detailed in the other thread here. I failed. http://forum.dlang.org/thread/l34lei$255v$1...@digitalmars.com That conversation started out from the D/Objective-C conversations. To have ARC in D and be compatible with the one in Objective-C you don't have many choices. I'm not sure but I don't think your proposal was not compatible with ARC in Objective-C. -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 2014-04-23 09:50:57 +, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com said: On Tuesday, 22 April 2014 at 19:42:20 UTC, Michel Fortin wrote: Objective-C isn't memory safe because it lets you play with raw pointers too. If you limit yourself to ARC-managed pointers (and avoid undefined behaviours inherited from C) everything is perfectly memory safe. I'm not convinced that it is safe in multi-threaded mode. How does ARC deal with parallell reads and writes from two different threads? IIRC the most common implementations deals with read/read and write/write, but read/write is too costly? The answer is that in the general case you should protect reads and writes to an ARC pointer with locks. Otherwise the counter risk being getting out of sync and later you'll get corruption somewhere. There are atomic properties which are safe to read and write from multiple threads. Internally they use the @synchronized keyword on the object. But since there's no 'shared' attribute in Objective-C, you can't go very far if you wanted the compiler to check things for memory safety. That said, if you assume a correct implementation of the NSCopying protocol (deep copying), objects following that protocol would be safe to pass through a std.concurrency-like interface. In all honesty, I'm not that impressed with the multithreading protections in D either. It seems you so often have to bypass the type system to make something useful that it doesn't appear very different from not having them. And don't get me started with synchronized classes... -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: DIP60: @nogc attribute
On 2014-04-23 04:33:00 +, Walter Bright newshou...@digitalmars.com said: On 4/22/2014 12:42 PM, Michel Fortin wrote: On 2014-04-22 19:02:05 +, Walter Bright newshou...@digitalmars.com said: Memory safety is not a strawman. It's a critical feature for a modern language, and will become ever more important. What you don't seem to get is that ARC, by itself, is memory-safe. I repeatedly said that it is not memory safe because you must employ escapes from it to get performance. It wasn't that clear to me you were saying that, but now it makes sense. In Objective-C, the performance-sensitive parts are going to be implemented in C, that's true. But that's rarely going to be more than 5% of your code, and probably only a few isolated parts where you're using preallocated memory blocks retained by ARC while you're playing with the content. If you're writing something that can't tolerate a GC pause, then it makes perfect sense to make this performance-critical code unsafe so you can write the remaining 95% of your app in a memory-safe environment with no GC pause. D on the other hand forces you to have those GC pauses or have no memory management at all. It's a different tradeoff and it isn't suitable everywhere, but I acknowledge it makes it easier to make performance-sensitive code @safe, something that'd be a shame to lose. Objective-C isn't memory safe because it lets you play with raw pointers too. If you limit yourself to ARC-managed pointers (and avoid undefined behaviours inherited from C) everything is perfectly memory safe. Allow me to make it clear that IF you never convert an ARC reference to a raw pointer in userland, I agree that it is memory safe. But this is not practical for high performance code. Framing the problem this way makes it easier to find a solution. I wonder, would it be acceptable if ARC was used everywhere by default but could easily be disabled inside performance-sensitive code by allowing the user choose between safe GC-based memory management or unsafe manual memory management? I have an idea that'd permit just that. Perhaps I should write a DIP about it. I'm pretty confident that had I continued my work on D/Objective-C we'd now be able to interact with Objective-C objects using ARC in @safe code. I was planning for that. Objective-C actually isn't very far from memory safety now that it has ARC, it just lacks the @safe attribute to enable compiler verification. I wish you would continue that work! I wish I had the time too. -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: DIP60: @nogc attribute
On Wed, 23 Apr 2014 05:14:44 -0400, Manu via Digitalmars-d digitalmars-d@puremagic.com wrote: On 23 April 2014 04:28, Steven Schveighoffer via Digitalmars-d ARC does not equal guaranteed memory safety. So NO, it cannot replace the GC for D @safe code. That doesn't make it useless. Why not? Assuming that direct access to the refcount is not @safe, why would ARC be unsafe? What makes it less safe than the GC? Arguably, it is safe, as long as you only use ARC pointers. I don't know that I would ever want or use that in D (or even Objective-C). So it's not that it's not safe, it's that it cannot be a drop-in-replacement for the GC in existing D @safe code. For example, you could never use slices or ranges, or these would have to be rewritten to keep references to the full object. -Steve
Re: DIP60: @nogc attribute
On Wed, 23 Apr 2014 02:11:38 -0400, Jacob Carlborg d...@me.com wrote: On 22/04/14 20:48, Steven Schveighoffer wrote: I mean not like I can't because I don't want to or don't have time, but can't as in I lack the skill set :) It's interesting to debate, and I get the concepts, but I am not a CPU/cache guy, and these things are really important to get right for performance, since ref counting would be used frequently. That's the worst kind of excuses :) I don't remember the last time I started working on a project and know what I was doing/had the right skill set. I mean, that's how you learn. Sure, but there are things I CAN do with my limited time, that I do have the expertise for. I've already been schooled by the likes of you and Michel Fortin on my knowledge of ref counting implementation. BTW, this is how RedBlackTree (dcollections) came into existence, I had no idea what I was doing, just the API that I wanted (and back then, I had more time). The code is actually a slightly-hand-optimized copy of my CLR book's red-black algorithm. -Steve
Re: DIP60: @nogc attribute
On 2014-04-23 17:57, Steven Schveighoffer wrote: Sure, but there are things I CAN do with my limited time, that I do have the expertise for. I've already been schooled by the likes of you and Michel Fortin on my knowledge of ref counting implementation. That's completely different. I've felt the same for a long time. Instead of working on the compiler I built tools and libraries for D. Then I finally couldn't keep my hands off and now I have D/Objective-C working for 64bit :) -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 4/23/2014 6:10 AM, Jacob Carlborg wrote: That conversation started out from the D/Objective-C conversations. To have ARC in D and be compatible with the one in Objective-C you don't have many choices. I'm not sure but I don't think your proposal was not compatible with ARC in Objective-C. Too many double negatives for me to be sure what you're saying. But it is clear to me that with Michel's experience with ARC in iOS combined with Manu's enthusiasm for it suggests that they are the right team to come up with a workable proposal, where mine failed.
Re: DIP60: @nogc attribute
On Monday, 21 April 2014 at 23:02:54 UTC, Walter Bright wrote:
There is JUST NO WAY that:
struct RefCount {
T* data;
int* count;
}
This is actually quite efficient compared to the standard
NSObject which uses a hashtable for refcounting:
http://www.opensource.apple.com/source/objc4/objc4-551.1/runtime/NSObject.mm
http://www.opensource.apple.com/source/objc4/objc4-551.1/runtime/llvm-DenseMap.h
This is how Core Foundation does it:
http://www.opensource.apple.com/source/CF/CF-855.11/CFRuntime.c
Pretty longwinded:
CFTypeRef CFRetain(CFTypeRef cf) {
if (NULL == cf) { CRSetCrashLogMessage(*** CFRetain() called
with NULL ***); HALT; }
if (cf) __CFGenericAssertIsCF(cf);
return _CFRetain(cf, false);
}
static CFTypeRef _CFRetain(CFTypeRef cf, Boolean tryR) {
uint32_t cfinfo = *(uint32_t *)(((CFRuntimeBase
*)cf)-_cfinfo);
if (cfinfo 0x80) { // custom ref counting for object
...stuff deleted…
refcount(+1, cf);
return cf;
}
…lots of stuff deleted…
return cf;
}
Re: DIP60: @nogc attribute
On 21/04/14 19:49, Frustrated wrote:
Not quite. AST macros simply transform code. Attributes attach meta data
to code. While I'm sure there is some overlap they are not the same.
Unless AST macros have the ability to arbitrary add additional
contextual information to meta code then they can't emulate attributes.
I'm not saying we should emulate attributes, we already have those.
BTW, I'm pretty sure they could be implemented with macros. Just return
the exact same AST that was passed in, but replace the top AST node with
a node that is a subclass that adds the data for the UDA.
E.g., Suppose you have D with AST macros but not attributes, how can you
add them?
In the dip, you have
macro attr (Context context, Declaration decl)
{
auto attrName = decl.name;
auto type = decl.type;
return [
private $decl.type _$decl.name;
$decl.type $decl.name ()
{
return _$decl.name;
}
$decl.type $decl.name ($decl.type value)
{
return _$decl.name = value;
}
];
}
class Foo
{
@attr int bar;
}
but attr is not an attribute. It is an macro. @attr converts the int
bar field into a private setter and getter. This has nothing to do with
attributes.
Sure it does. @nogc could be implemented with AST macros.
@nogc void foo ()
{
new Object;
}
macro nogc (Context context, Declaration decl)
{
if (containsGCAllocation(decl))
context.compiler.error(decl.name ~ marked with @nogc performs
GC allocations);
return decl;
}
(just cause you use the attr word and the @ symbol doesn't make it an
attribute)
I don't see how you could ever add attributes to D using AST macros
above unless the definition of an AST macro is modified. [Again,
assuming D didn't have attributes in the first place]
This does not mean that AST macros could not be used to help define the
generalized attributes though.
What I am talking about is instead of hard coding attributes in the
compiler, one abstracts and generalizes the code so that any attribute
could be added in the future with minimal work.
It would simply require one to add the built in attributes list, add the
attribute grammar(which is used to reduce compound attributes), add any
actions that happen when the attribute is used in code.
e.g.,
builtin_attributes = {
{pureness, pure, !pure/impure,
attr = any(attr, impure) = impure
attr = all(attr, pure) = pure
}
{gc, gc, !gc/nogc,
attr = any(attr, gc) = gc
attr = all(attr, nogc) = nogc
}
etc... }
notices that pureness and gc have the same grammatical rules. Code would
be added to handle the pureness and gc attributes when they are come
across for optimization purposes.
The above syntax is just made up and pretty bad but hopefully not too
difficult to get the bigger picture.
Every new built in attribute would just have to be added to the list
above(easy) and code that uses it for whatever purpose would be added in
the code where it belongs.
User define attributes essentially would make the attributes list above
dynamic allowing the user to add to it. The compiler would only be told
how to simplify the attributes using the grammar and would do so but
would not have any code inserted because there is no way for the user to
hook into the compiler properly(I suppose it could be done if the
compiler was written in an oop like way).
The AST macros would provide a way to hook into the compiler. We already
have a way to define attributes, that is, UDA's. What is missing is a
way to add semantic meanings the UDA's, that is where macros come in.
--
/Jacob Carlborg
Re: DIP60: @nogc attribute
On 22/04/14 01:02, Walter Bright wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. This makes it simply not acceptable for D in the general case. It works with iOS because iOS allows all kinds of (unsafe) ways to escape it, and it must offer those ways because it is not performant. So does D. That's why there is @safe, @trusted and @system. What is the unsafe part of ARC anyway? -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 4/21/2014 11:51 PM, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: This is actually quite efficient compared to the standard NSObject which uses a hashtable for refcounting: It's not efficient compared to pointers.
Re: DIP60: @nogc attribute
On 4/22/2014 12:11 AM, Jacob Carlborg wrote: On 22/04/14 01:02, Walter Bright wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. This makes it simply not acceptable for D in the general case. It works with iOS because iOS allows all kinds of (unsafe) ways to escape it, and it must offer those ways because it is not performant. So does D. That's why there is @safe, @trusted and @system. What is the unsafe part of ARC anyway? As I said, it is when it is bypassed for performance reasons.
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 09:02:21 UTC, Walter Bright wrote: On 4/22/2014 12:11 AM, Jacob Carlborg wrote: On 22/04/14 01:02, Walter Bright wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. This makes it simply not acceptable for D in the general case. It works with iOS because iOS allows all kinds of (unsafe) ways to escape it, and it must offer those ways because it is not performant. So does D. That's why there is @safe, @trusted and @system. What is the unsafe part of ARC anyway? As I said, it is when it is bypassed for performance reasons. A system that is automatically safe but can be manually managed for extra performance. That sounds very D-ish.
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 09:40:15 UTC, Ola Fosheim Grøstad
wrote:
if( CAS_SET_BIT(ref+32,THREADID)==THREADID ){
Make that:
if( CAS_SET_BIT(ref+32,THREADID) == (1THREADID) ){
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 09:01:20 UTC, Walter Bright wrote:
On 4/21/2014 11:51 PM, Ola Fosheim Grøstad
ola.fosheim.grostad+dl...@gmail.com wrote:
This is actually quite efficient compared to the standard
NSObject which uses a
hashtable for refcounting:
It's not efficient compared to pointers.
It isn't efficient compared to pointers if you use a blind ARC
implementation. If you use ARC to track ownership of regions then
it can be efficient.
The real culprit is multithreading, it can be resolved though. If
you put the counters on cachelines that are local to the thread,
either by offset or TLS.
E.g. pseudocode for 8 refcounted pointers on 4 threads with 32
bytes cachelines could be something along the lines of:
struct {
func* destructor[8]; // cacheline -1
void* ptr[8]; //cacheline 0
uint bitmask[8]; //cacheline 1
int refcount[8*4] ; //cacheline2-6 initialized to -1
}
THREADOFFSET = (THREADID+2)*32
retain(ref){ //ref is a pointer to cacheline 0
if ( increment(ref+THREADOFFSET) == 0) {
if( CAS_SET_BIT(ref+32,THREADID)==THREADID ){
HALT_DESTRUCTED()
}
}
}
release(ref){
if( decrement(ref+THREADOFFSET)0 ){
if( CAS_CLR_BIT(ref+32,THREADID)==0){
call_destructor(ref-32,*ref);
}
}
}
Re: DIP60: @nogc attribute
On Thursday, 17 April 2014 at 19:51:38 UTC, Walter Bright wrote: On 4/17/2014 10:41 AM, Dicebot wrote: On Thursday, 17 April 2014 at 16:57:32 UTC, Walter Bright wrote: With current limitations @nogc is only useful to verify that embedded code which does not have GC at all does not use any GC-triggering language features before it comes to weird linker errors / rt-asserts. But that does not work good either because of next problem: Remember that @nogc will be inferred for template functions. That means that whether it is @nogc or not will depend on its arguments being @nogc, which is just what is needed. No, it looks like I have stated that very wrong because everyone understood it in completely opposite way. What I mean is that `put()` is NOT @nogc and it still should work. Same as weakly pure is kind of pure but allowed to mutate its arguments, proposed weakly @nogc can only call GC via functions directly accessible from its arguments. I don't see value for this behavior. It turns out to have enormous value. I will explain this in my DConf talk. A little preview: Almost all of our code at Sociomantic obeys this behaviour, and it's probably the most striking feature of our codebase. By almost all I mean probably 90% of our code, including all of our libraries. Not just the 5% - 10% that could marked as @nogc according to your DIP. The key property it ensures is, if you make N calls to the function, the number of GC allocations is in O(1). We don't care if makes 0 allocations or 17. We're not really interested in whether a function uses the GC or not, since most interesting functions do need to do some memory allocation. Ideally, we'd want an attribute which could applied to *all* of Phobos, except for some convenience functions. We have no interest in library code which doesn't behave in that way.
Re: DIP60: @nogc attribute
On Mon, 21 Apr 2014 17:52:30 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: Here are two very good reasons to avoid extensive ref-counting: 1. transactional memory ( you don't want a lock on two reads ) 2. cache coherency ( you don't want barriers everywhere ) Betting everything on ref counting is the same as saying no to upcoming CPUs. IMO that means ARC is DOA. It might be useful for some high level objects… but I don't understand why one would think that it is a good low level solution. It is a single-threaded solution. Single threaded ARC can go a long way in D. We statically know whether data is shared or not. -Steve
Re: DIP60: @nogc attribute
On Mon, 21 Apr 2014 19:02:53 -0400, Walter Bright
newshou...@digitalmars.com wrote:
On 4/21/2014 1:29 PM, Steven Schveighoffer wrote:
I think you are misunderstanding something. This is not for a pervasive
ARC-only, statically guaranteed system. The best example he gives (and
I agree
with him) is iOS. Just look at the success of iOS, where the entire OS
API is
based on ARC (actually RC, with an option for both ARC and manual, but
the
latter is going away). If ARC was so bad, the iOS experience would
show it.
You may have doubts, but I can assure you I can build very robust and
performant
code with ARC in iOS.
The thing is, with iOS ARC, it cannot be statically guaranteed to be
memory safe.
So?
This makes it simply not acceptable for D in the general case.
Because it can't live beside all the other unsafe code in D? I don't get
it...
It works with iOS because iOS allows all kinds of (unsafe) ways to
escape it, and it must offer those ways because it is not performant.
I think we're officially going in circles here.
Kinda sorta memory safe, mostly memory safe, etc., is not a static
guarantee.
There is JUST NO WAY that:
struct RefCount {
T* data;
int* count;
}
is going to be near as performant as:
T*
Again with the straw man!
1. A dereference requires two indirections. Cache performance, poof!
2. A copy requires two indirections to inc, two indirections to dec, and
an exception unwind handler for dec.
3. Those two word structs add to memory consumption.
Consider the straw man destroyed :)
As you pointed out, performant code is going to have to cache the data*
value. That cannot be guaranteed memory safe.
I can't reconcile agreeing that ARC isn't good enough to be pervasive
with
compiler technology eliminates unnecessary ARC overhead.
It's pretty pervasive on iOS. ARC has been around since iOS 4.3 (circa
2011).
Pervasive means for all pointers. This is not true of iOS. It's fine
for iOS to do a half job of it, because the language makes no
pretensions about memory safety. It is not fine for D to replace a
guaranteed memory safe system with an unsafe,
hope-your-programmers-get-it-right, solution.
Totally agree, which is why nobody is saying that.
-Steve
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 13:11:55 UTC, Steven Schveighoffer wrote: Single threaded ARC can go a long way in D. Not without changing language semantics/mechanisms? We statically know whether data is shared or not. I don't understand how you can know this when you allow foreign function invocation. Even thread local globals can leak into another thread by mistake, unnoticed? Meaning random crashes that are hard to debug?
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 09:29:28 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Tuesday, 22 April 2014 at 13:11:55 UTC, Steven Schveighoffer wrote: Single threaded ARC can go a long way in D. Not without changing language semantics/mechanisms? We statically know whether data is shared or not. I don't understand how you can know this when you allow foreign function invocation. Can you explain this? Even thread local globals can leak into another thread by mistake, unnoticed? Meaning random crashes that are hard to debug? By mistake? How? -Steve
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 13:39:54 UTC, Steven Schveighoffer wrote: On Tue, 22 Apr 2014 09:29:28 -0400, Ola Fosheim Grøstad Can you explain this? When you use a C/C++ framework you don't know what happens to the pointers you hand to it. You also don't know which threads call your D-functons from that framework. (Assuming the framework is multi-threaded). To know this you are required to know the internals of the framework you are utilizing or inject runtime guards into your D functions? By mistake? How? By insertion into a global datastructure that happens at a lower layer than the higher level you allocate on.
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 09:48:28 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Tuesday, 22 April 2014 at 13:39:54 UTC, Steven Schveighoffer wrote: On Tue, 22 Apr 2014 09:29:28 -0400, Ola Fosheim Grøstad Can you explain this? When you use a C/C++ framework you don't know what happens to the pointers you hand to it. Those are typically well-documented, but yes, you rely on insider knowledge. You can mark C functions with the appropriate attributes so the D compiler can enforce this for you. I think we should make reference counting work, as long as you don't call mischievous library code. We take a similar approach to other safety aspects of D. You also don't know which threads call your D-functons from that framework. (Assuming the framework is multi-threaded). To know this you are required to know the internals of the framework you are utilizing or inject runtime guards into your D functions? Or just mark those objects sent into the framework as shared. Having multi-threaded RC isn't bad, just not as efficient. One thing that would be nice is to allow moving a data pointer from one thread to another. In other words, as long as your data is contained, it can pass from one thread to another, and still be considered unshared. I think sooner or later, we are going to have to figure that one out. By mistake? How? By insertion into a global datastructure that happens at a lower layer than the higher level you allocate on. I think this is what you are talking about above, or is there something else? -Steve
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 14:07:47 UTC, Steven Schveighoffer wrote: know this you are required to know the internals of the framework you are utilizing or inject runtime guards into your D functions? Or just mark those objects sent into the framework as shared. Having multi-threaded RC isn't bad, just not as efficient. Actually, when I think of it, guards probably would be cheap. All you have to do is to store the thread-context-pointer-register into a global when the thread starts up. Then just do a simple if-test at the function invocation. (assuming the register doesn't change over time). Actually, it could be done as a self-modifying pass at startup… That would make it a register test against an immediate value, no memory buss implications. One thing that would be nice is to allow moving a data pointer from one thread to another. In other words, as long as your data is contained, it can pass from one thread to another, and still be considered unshared. Yes, that sounds plausible. I think this is what you are talking about above, or is there something else? You are right :). Ola.
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 06:51:40 UTC, Ola Fosheim Grøstad
wrote:
On Monday, 21 April 2014 at 23:02:54 UTC, Walter Bright wrote:
There is JUST NO WAY that:
struct RefCount {
T* data;
int* count;
}
This is actually quite efficient compared to the standard
NSObject which uses a hashtable for refcounting:
iOS now on 64-bit processors doesn't necessarily use a hashtable
for refcounting. Basically, only 33 bits of the 64-bit pointer
are used to actually refer to an address, then 19 of the
remaining bits are used to store an inline reference count. Only
if the inline reference count exceeds these 19 bits (very rare)
do they switch to using a hashtable. It was one of the large
benefits of switching to ARM64 for the iPhone 5.
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 16:58:23 UTC, Kapps wrote: iOS now on 64-bit processors doesn't necessarily use a hashtable for refcounting. Basically, only 33 bits of the 64-bit pointer are used to actually refer to an address, then 19 of the remaining bits are used to store an inline reference count. I am sure you are right, but how does that work? Do you have a link?
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 13:22:19 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Tuesday, 22 April 2014 at 16:58:23 UTC, Kapps wrote: iOS now on 64-bit processors doesn't necessarily use a hashtable for refcounting. Basically, only 33 bits of the 64-bit pointer are used to actually refer to an address, then 19 of the remaining bits are used to store an inline reference count. I am sure you are right, but how does that work? Do you have a link? I think what he's saying is the 19 bits are an offset into the global ref count container. Some sentinel value means lookup the pointer in a hashtable. -Steve
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 17:22:21 UTC, Ola Fosheim Grøstad wrote: On Tuesday, 22 April 2014 at 16:58:23 UTC, Kapps wrote: iOS now on 64-bit processors doesn't necessarily use a hashtable for refcounting. Basically, only 33 bits of the 64-bit pointer are used to actually refer to an address, then 19 of the remaining bits are used to store an inline reference count. I am sure you are right, but how does that work? Do you have a link? https://www.mikeash.com/pyblog/friday-qa-2013-09-27-arm64-and-you.html Ctrl +F Repurposed isa Pointer Details about isa structure: http://www.sealiesoftware.com/blog/archive/2013/09/24/objc_explain_Non-pointer_isa.html
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 13:35:20 -0400, Steven Schveighoffer schvei...@yahoo.com wrote: On Tue, 22 Apr 2014 13:22:19 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Tuesday, 22 April 2014 at 16:58:23 UTC, Kapps wrote: iOS now on 64-bit processors doesn't necessarily use a hashtable for refcounting. Basically, only 33 bits of the 64-bit pointer are used to actually refer to an address, then 19 of the remaining bits are used to store an inline reference count. I am sure you are right, but how does that work? Do you have a link? I think what he's saying is the 19 bits are an offset into the global ref count container. Some sentinel value means lookup the pointer in a hashtable. Sorry, I was wrong. And so was Kapps, according to the article he referenced. The 33 bits of the 64 bit pointer are used to point at a *class*, via the object's isa member. The other 19 bits can be used for ref counting. I mistakenly thought he meant the pointer to the object. This is kind of weird though. Why do it this way? If there is such an advantage to having the ref count inside the object (and I don't disagree with that), why didn't they do that before? Surely adding another 32-bit field wouldn't have killed the runtime. Even doing it the way they have seems unnecessarily complex, given that iOS 64-bit was brand new. -Steve
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 17:39:53 UTC, Kapps wrote: https://www.mikeash.com/pyblog/friday-qa-2013-09-27-arm64-and-you.html Ctrl +F Repurposed isa Pointer Ah, ok, the refcount is embedded in the class-table pointer.
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 17:52:28 UTC, Steven Schveighoffer wrote: Even doing it the way they have seems unnecessarily complex, given that iOS 64-bit was brand new. I dislike this too… The only reason I can think of is that Apple themselves have code for OS-X that is optimized 64 bit code and that will break if they add another field? The lower bit is used for compatibility mode. Hmm…
Re: DIP60: @nogc attribute
On 4/22/2014 6:18 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 19:02:53 -0400, Walter Bright newshou...@digitalmars.com wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. So? If you see no value in static guarantees of memory safety, then what can I say?
Re: DIP60: @nogc attribute
On 4/22/2014 2:20 AM, John Colvin wrote: A system that is automatically safe but can be manually managed for extra performance. That sounds very D-ish. Needing to write @system code with a GC to get performance is rare in D. It's normal in O-C, as has been pointed out here a couple times.
Re: DIP60: @nogc attribute
On 4/22/2014 6:18 AM, Steven Schveighoffer wrote: Again with the straw man! If you really believe you can make a performant ARC system, and have it be memory safe, feel free to write a complete proposal on it.
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 14:12:17 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/22/2014 6:18 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 19:02:53 -0400, Walter Bright newshou...@digitalmars.com wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. So? If you see no value in static guarantees of memory safety, then what can I say? Seriously, the straw man arguments have to stop. There is plenty of valuable D code that is not guaranteed memory safe. For example, druntime. ARC does not equal guaranteed memory safety. So NO, it cannot replace the GC for D @safe code. That doesn't make it useless. -Steve
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 14:15:35 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/22/2014 6:18 AM, Steven Schveighoffer wrote: Again with the straw man! If you really believe you can make a performant ARC system, and have it be memory safe, feel free to write a complete proposal on it. I hope you can understand that from this discussion, I'm not to motivated to devote time on it. Not that I could do it anyway :) Generally, when investing a lot of time and energy into something, you want to make sure the market is there first... -Steve
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 18:38:28 UTC, Steven Schveighoffer wrote: I hope you can understand that from this discussion, I'm not to motivated to devote time on it. Not that I could do it anyway :) Do it anyway. This is such a fun topic and many would be entertained by the ensuing excruciating inquisitorial process.
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 14:43:04 -0400, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Tuesday, 22 April 2014 at 18:38:28 UTC, Steven Schveighoffer wrote: I hope you can understand that from this discussion, I'm not to motivated to devote time on it. Not that I could do it anyway :) Do it anyway. I mean not like I can't because I don't want to or don't have time, but can't as in I lack the skill set :) It's interesting to debate, and I get the concepts, but I am not a CPU/cache guy, and these things are really important to get right for performance, since ref counting would be used frequently. -Steve
Re: DIP60: @nogc attribute
On 2014-04-22 17:52:27 +, Steven Schveighoffer schvei...@yahoo.com said: Even doing it the way they have seems unnecessarily complex, given that iOS 64-bit was brand new. Perhaps it's faster that way due to some caching effect. Or perhaps it's to be able to have static constant string objects in the readonly segments. Apple could always change their mind and add another field for the reference count. The Modern runtime has non-fragile classes, so you can change the base class layout breaking ABI compatibility. -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: DIP60: @nogc attribute
On Tuesday, 22 April 2014 at 18:48:04 UTC, Steven Schveighoffer wrote: I mean not like I can't because I don't want to or don't have time, but can't as in I lack the skill set :) It's interesting to debate, and I get the concepts, but I am not a CPU/cache guy, and these things are really important to get right for performance, since ref counting would be used frequently. I think RC performance unfortunately is very hardware dependent. So, it involves testing, benchmarking etc… Still, I think these discussions are good because they are opportunities to look at what others do (like Objective-C) which is educational (at least for me). The trouble I have with ref counting is that the future of HW is uncertain, but that can be held against GC too. Take for instance Phi with 80 cores and a crossbar interconnects (?), how does that affect memory management? Unfortunately I have no personal experience with Phi so my take on it is rather vague… But if ARC takes a long time to get right I think one should consider effects of upcoming HW when considering for/against. Maybe Go's CSP model is better for Phi. Maybe not. I don't know, but I think it is an interesting topic, because 4-8 cores is not going to be enough in the next decade. I think…
Re: DIP60: @nogc attribute
On 4/22/2014 11:28 AM, Steven Schveighoffer wrote: On Tue, 22 Apr 2014 14:12:17 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/22/2014 6:18 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 19:02:53 -0400, Walter Bright newshou...@digitalmars.com wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. So? If you see no value in static guarantees of memory safety, then what can I say? Seriously, the straw man arguments have to stop. There is plenty of valuable D code that is not guaranteed memory safe. Memory safety is not a strawman. It's a critical feature for a modern language, and will become ever more important. For example, druntime. Nobody expects a GC's guts to be guaranteed memory safe. But they do expect the interface to it to be memory safe, and using GC allocated data to be memory safe, and being able to write performant memory safe code. And by memory safe, I don't mean hand verified. I mean machine verified.
Re: DIP60: @nogc attribute
On 4/22/2014 11:38 AM, Steven Schveighoffer wrote: Generally, when investing a lot of time and energy into something, you want to make sure the market is there first... Ironic, considering that nobody but me believed there was a market for D before it existed :-) I do believe there is a market for ARC in D. What I don't believe are the various claims about how insignificant its costs are, and I'm not so willing to give up on memory safety.
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 15:02:05 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/22/2014 11:28 AM, Steven Schveighoffer wrote: On Tue, 22 Apr 2014 14:12:17 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/22/2014 6:18 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 19:02:53 -0400, Walter Bright newshou...@digitalmars.com wrote: The thing is, with iOS ARC, it cannot be statically guaranteed to be memory safe. So? If you see no value in static guarantees of memory safety, then what can I say? Seriously, the straw man arguments have to stop. There is plenty of valuable D code that is not guaranteed memory safe. Memory safety is not a strawman. It's a critical feature for a modern language, and will become ever more important. No, a straw man argument is when you imply that I am arguing from a position that is similar to my actual position, but obviously flawed. Then proceed to attack the straw man. Example: A: Sunny days are good. B: If all days were sunny, we'd never have rain, and without rain, we'd have famine and death. At no time did I ever say I see no value in static guarantees of memory safety. But I also see value in ref counting for performance and memory purposes in NON memory-safe code. -Steve
Re: DIP60: @nogc attribute
On Tue, 22 Apr 2014 15:10:31 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/22/2014 11:38 AM, Steven Schveighoffer wrote: Generally, when investing a lot of time and energy into something, you want to make sure the market is there first... Ironic, considering that nobody but me believed there was a market for D before it existed :-) Sure but I'm not prepared to generate a new language over this. If you won't accept it, there's no point in making it. In other words, YOU are the market ;) *disclaimer* I am in no shape to actually make such a proposal/change, and probably it would come out horribly if I tried, implementation-wise. I simply am trying to convince you that it would be a valuable addition to D so others may see an opportunity there. I do believe there is a market for ARC in D. What I don't believe are the various claims about how insignificant its costs are, and I'm not so willing to give up on memory safety. You don't have to. Just make ARC not @safe. -Steve
Re: DIP60: @nogc attribute
On 2014-04-22 19:02:05 +, Walter Bright newshou...@digitalmars.com said: Memory safety is not a strawman. It's a critical feature for a modern language, and will become ever more important. What you don't seem to get is that ARC, by itself, is memory-safe. Objective-C isn't memory safe because it lets you play with raw pointers too. If you limit yourself to ARC-managed pointers (and avoid undefined behaviours inherited from C) everything is perfectly memory safe. I'm pretty confident that had I continued my work on D/Objective-C we'd now be able to interact with Objective-C objects using ARC in @safe code. I was planning for that. Objective-C actually isn't very far from memory safety now that it has ARC, it just lacks the @safe attribute to enable compiler verification. -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: DIP60: @nogc attribute
On 4/22/2014 12:42 PM, Michel Fortin wrote: On 2014-04-22 19:02:05 +, Walter Bright newshou...@digitalmars.com said: Memory safety is not a strawman. It's a critical feature for a modern language, and will become ever more important. What you don't seem to get is that ARC, by itself, is memory-safe. I repeatedly said that it is not memory safe because you must employ escapes from it to get performance. Objective-C isn't memory safe because it lets you play with raw pointers too. If you limit yourself to ARC-managed pointers (and avoid undefined behaviours inherited from C) everything is perfectly memory safe. Allow me to make it clear that IF you never convert an ARC reference to a raw pointer in userland, I agree that it is memory safe. But this is not practical for high performance code. I'm pretty confident that had I continued my work on D/Objective-C we'd now be able to interact with Objective-C objects using ARC in @safe code. I was planning for that. Objective-C actually isn't very far from memory safety now that it has ARC, it just lacks the @safe attribute to enable compiler verification. I wish you would continue that work!
Re: DIP60: @nogc attribute
On 4/22/2014 4:00 AM, Don wrote: It turns out to have enormous value. I will explain this in my DConf talk. A little preview: Almost all of our code at Sociomantic obeys this behaviour, and it's probably the most striking feature of our codebase. By almost all I mean probably 90% of our code, including all of our libraries. Not just the 5% - 10% that could marked as @nogc according to your DIP. The key property it ensures is, if you make N calls to the function, the number of GC allocations is in O(1). We don't care if makes 0 allocations or 17. I don't really understand how dicebot's proposal ensures this property. I guess it'll have to wait until Dconf! We're not really interested in whether a function uses the GC or not, since most interesting functions do need to do some memory allocation. Ideally, we'd want an attribute which could applied to *all* of Phobos, except for some convenience functions. We have no interest in library code which doesn't behave in that way.
Re: DIP60: @nogc attribute
Timon Gehr: In which case? In case some version of LDC2 is able to avoid the heap allocation using full optimizations? :o) Please take a look, I have just added one more note in the optimizations section: http://wiki.dlang.org/DIP60#Behaviour_in_presence_of_optimizations Bye, bearophile
Re: DIP60: @nogc attribute
On Fri, 18 Apr 2014 20:17:59 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/18/2014 5:30 AM, Steven Schveighoffer wrote: Absolutely not, the compiler knows whether the count needs to be incremented, I don't need to know. But there are manual escapes from it, meaning you need to know to use them: unsigned char *vdata = data.data; // process vdata I am skeptical that the O-C compiler is sufficiently smart to make ARC on all pointers practical. That snippet pretty much proves it. I finally understand what you are saying. Yes, when my code devolves into C-land, I have non-ARC pointers. This means I cannot store these pointers somewhere if I don't keep a pointer to the object that owns them. This is indeed manual memory management. But, and this is a big but, I DON'T have to care about memory management as long as I only store pointers to objects, aside from the stack. Objects themselves can store non-ARC pointers and manage them privately, but not allow arbitrary setting/getting of pointers. This is not very common. A great example of where I use arbitrary non-GC pointers, is an object that stores a network packet. unix sockets work just fine in networking and iOS, so I do all of my networking with socket, send, recv, etc. These functions take unmanaged char * pointers. So an object that stores a single packet does this: 1. read the packet header into a structure overlay. Figure out the length (this is a TCP packet). 2. malloc a char array with the proper length, read in the packet. 3. verify the packet is correct, if not, free the buffer. 4. If it's correct, digest the packet (storing fields from the packet data itself, byteswap, etc.). Then store the data into an NSData object, giving it ownership of the buffer. All this happens within a factory method. I never directly store the char * array inside the object or elsewhere, it gets wrapped into an NSData object, which will then free the data once the object is destroyed. You must be conscious about memory management, because after all, Objective-C *is* C, and you are allowed to do stupid things just like in C. But you CAN have a certain set of rules, and as long as you follow those rules, you will not have to deal with memory management. Total replacement of GC with ARC in D will: This is the wrong straw-man, I'm not advocating for this at all. 1. Be a massive change that will break most every D program 2. Require people to use unsafe escapes to recover lost performance 3. Require multiple pointer types This isn't what I had in mind anyway. I envisioned an ARC system like Objective-C, where the type pointed at defines whether it's ARC or GC. For instance, GC would be *required* for array appending in D, because arrays cannot be ARC-managed. You would need a wrapper type for ARC, with it's own semantics. 4. Will not be memory safe (see (2)) 5. Require the invention of optimization technology that doesn't exist Besides being a tautology, what does this mean? 6. Become more or less ABI incompatible with C without a long list of caveats and translations Again, this is based on your straw-man which I don't advocate for. The change I had in mind was much less invasive. A much more tractable idea is to implement something like C++'s shared_ptr as a library type, with usage strategies paralleling C++'s (and yes, use of shared_ptr would be unsafe). shared_ptr would work (and I've used that extensively too, it's awesome for C++), but I feel it's a much less effective solution than a compiler-augmented system that can optimize away needless increment/decrements. Note that shared_ptr would never be able to handle D's slice appending either. -Steve
Re: DIP60: @nogc attribute
On Sunday, 20 April 2014 at 14:38:47 UTC, Frustrated wrote: How bout this! Why not allow one to define their own attributes from a generalized subset and then define a few standard ones like @nogc. Sounds like you want AST macros. Have a look at this http://wiki.dlang.org/DIP50#Declaration_macros -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On 4/21/2014 5:00 AM, Steven Schveighoffer wrote: Total replacement of GC with ARC in D will: This is the wrong straw-man, I'm not advocating for this at all. Many are when they advocate ARC for D. 4. Will not be memory safe (see (2)) 5. Require the invention of optimization technology that doesn't exist Besides being a tautology, what does this mean? 4. There is no language protection against misusing ARC, hence it cannot be mechanically verified to be memory safe. 5. Numerous posters here have posited that the overhead of ARC can be eliminated with a sufficiently smart compiler (which does not exist). Note that shared_ptr would never be able to handle D's slice appending either. I know. shared_ptr would, of course, be used at the specific discretion of the programmer. It would not be under the hood, and it would not be memory safe.
Re: DIP60: @nogc attribute
On Monday, 21 April 2014 at 16:45:15 UTC, Jacob Carlborg wrote:
On Sunday, 20 April 2014 at 14:38:47 UTC, Frustrated wrote:
How bout this!
Why not allow one to define their own attributes from a
generalized subset and then define a few standard ones like
@nogc.
Sounds like you want AST macros. Have a look at this
http://wiki.dlang.org/DIP50#Declaration_macros
--
/Jacob Carlborg
Not quite. AST macros simply transform code. Attributes attach
meta data to code. While I'm sure there is some overlap they are
not the same.
Unless AST macros have the ability to arbitrary add additional
contextual information to meta code then they can't emulate
attributes.
E.g., Suppose you have D with AST macros but not attributes, how
can you add them?
In the dip, you have
macro attr (Context context, Declaration decl)
{
auto attrName = decl.name;
auto type = decl.type;
return [
private $decl.type _$decl.name;
$decl.type $decl.name ()
{
return _$decl.name;
}
$decl.type $decl.name ($decl.type value)
{
return _$decl.name = value;
}
];
}
class Foo
{
@attr int bar;
}
but attr is not an attribute. It is an macro. @attr converts the
int bar field into a private setter and getter. This has
nothing to do with attributes.
(just cause you use the attr word and the @ symbol doesn't make
it an attribute)
I don't see how you could ever add attributes to D using AST
macros above unless the definition of an AST macro is modified.
[Again, assuming D didn't have attributes in the first place]
This does not mean that AST macros could not be used to help
define the generalized attributes though.
What I am talking about is instead of hard coding attributes in
the compiler, one abstracts and generalizes the code so that any
attribute could be added in the future with minimal work.
It would simply require one to add the built in attributes list,
add the attribute grammar(which is used to reduce compound
attributes), add any actions that happen when the attribute is
used in code.
e.g.,
builtin_attributes = {
{pureness, pure, !pure/impure,
attr = any(attr, impure) = impure
attr = all(attr, pure) = pure
}
{gc, gc, !gc/nogc,
attr = any(attr, gc) = gc
attr = all(attr, nogc) = nogc
}
etc... }
notices that pureness and gc have the same grammatical rules.
Code would be added to handle the pureness and gc attributes when
they are come across for optimization purposes.
The above syntax is just made up and pretty bad but hopefully not
too difficult to get the bigger picture.
Every new built in attribute would just have to be added to the
list above(easy) and code that uses it for whatever purpose would
be added in the code where it belongs.
User define attributes essentially would make the attributes list
above dynamic allowing the user to add to it. The compiler would
only be told how to simplify the attributes using the grammar and
would do so but would not have any code inserted because there is
no way for the user to hook into the compiler properly(I suppose
it could be done if the compiler was written in an oop like way).
In any case, because the compiler knows how to simplify UDA's by
using the provided grammar it makes UDA's much more powerful. CT
reflection and AST macros would make them way more powerful. The
ability to add hooks into the compiler would nearly give the user
the same power as the compiler writer. Of course, this would
probably lead to all kinds of compatibility problems in user code.
Basically, as of now, all we can do is define UDA's, we can't
define how they relate to each other(the grammar) nor what
happens when the exist(the behavior). There should not be any
real difference between UDA's and built in attributes(except
hooking, only because of complexity issues) and having a
generalized attribute system in the compiler would go a long way
to bridging the gap.
The main thing to take away from this is that *if* D had such an
attribute system, the gc/nogc issue would be virtually
non-existent. It would only take a few minutes to add a UDA
@gc/@nogc to D in user code. At least a few people would have
done it, and it's merits could be seen. Then it would be only a
matter of copy and pasting the definition of the attribute to
the compiler code and it would then become a built in making it
available for everyone.
At some point optimizations in the compiler could potentially be
added... just for fun. Not really the point of the attribute but
having it does provide such possibilities.
Also, with such a system, attributes can propagate from the
bottom up and inference makes it a lot easier.
E.g., atomic statements could be marked with attributes. Blocks
of statements can be marked but also inherit from the statements
and blocks they use.
With such a system, one could mark
Re: DIP60: @nogc attribute
On Mon, 21 Apr 2014 13:28:24 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/21/2014 5:00 AM, Steven Schveighoffer wrote: Total replacement of GC with ARC in D will: This is the wrong straw-man, I'm not advocating for this at all. Many are when they advocate ARC for D. Does that preclude you from accepting any kind of ARC for D? 5. Numerous posters here have posited that the overhead of ARC can be eliminated with a sufficiently smart compiler (which does not exist). You continue to speak in extremes. People are saying that the compiler can eliminate most of the needless ARC increments and decrements, not all of them. Compilers that do this do exist. Note that shared_ptr would never be able to handle D's slice appending either. I know. shared_ptr would, of course, be used at the specific discretion of the programmer. It would not be under the hood, and it would not be memory safe. Doesn't RefCounted do this already? -Steve
Re: DIP60: @nogc attribute
On 4/21/2014 10:57 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 13:28:24 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/21/2014 5:00 AM, Steven Schveighoffer wrote: Total replacement of GC with ARC in D will: This is the wrong straw-man, I'm not advocating for this at all. Many are when they advocate ARC for D. Does that preclude you from accepting any kind of ARC for D? No. My objection is to pervasive ARC, i.e. all gc is replaced with ARC, and it all magically works. 5. Numerous posters here have posited that the overhead of ARC can be eliminated with a sufficiently smart compiler (which does not exist). You continue to speak in extremes. People are saying that the compiler can eliminate most of the needless ARC increments and decrements, not all of them. Manu, for example, suggests it is good enough to make the overhead insignificant. I'm skeptical. Compilers that do this do exist. I can't reconcile agreeing that ARC isn't good enough to be pervasive with compiler technology eliminates unnecessary ARC overhead. I know. shared_ptr would, of course, be used at the specific discretion of the programmer. It would not be under the hood, and it would not be memory safe. Doesn't RefCounted do this already? Yes, but I haven't really looked into RefCounted.
Re: DIP60: @nogc attribute
Am Sun, 20 Apr 2014 08:19:45 +
schrieb monarch_dodra monarchdo...@gmail.com:
D static initialization doesn't work the same way. Everything is
initialized as the program is loaded, […]
Ah ok, it's all good then :)
Also, just doing this is good enough:
//
void foo() @nogc
{
static err = new Error(badthing happened);
if (badthing)
throw err;
}
//
It does require the message be known before hand, and not custom
built. But then again, where were you going to allocate the
message, and if allocated, who would clean it up?
That said, while the approach works, there could be issues with
re-entrance, or chaining exceptions.
Yes, we've discussed the issues with that approach in other
threads. At least this allows exceptions to be used at all.
--
Marco
Re: DIP60: @nogc attribute
On Mon, 21 Apr 2014 15:03:18 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/21/2014 10:57 AM, Steven Schveighoffer wrote: On Mon, 21 Apr 2014 13:28:24 -0400, Walter Bright newshou...@digitalmars.com wrote: On 4/21/2014 5:00 AM, Steven Schveighoffer wrote: Total replacement of GC with ARC in D will: This is the wrong straw-man, I'm not advocating for this at all. Many are when they advocate ARC for D. Does that preclude you from accepting any kind of ARC for D? No. My objection is to pervasive ARC, i.e. all gc is replaced with ARC, and it all magically works. With slicing, I don't think it's possible. Looking up the owner of a block for an INTERNAL pointer costs O(lg(n)), where n is all of your memory, because you must find out what block the pointer is accessing. Doing this during mark and sweep is bad enough, doing it for every ref count add or remove would be ghastly. ARC, and ref counting in general, is MUCH better implemented as having a reference count inside the object referenced. This means the type MUST BE AWARE of ref counting, or be wrapped in an aware type. Slices couldn't possibly do this correctly, since they do not point at the general block where all the info is maintained, but just at some memory. An ARC-aware slice would be fatter, or less performant. 5. Numerous posters here have posited that the overhead of ARC can be eliminated with a sufficiently smart compiler (which does not exist). You continue to speak in extremes. People are saying that the compiler can eliminate most of the needless ARC increments and decrements, not all of them. Manu, for example, suggests it is good enough to make the overhead insignificant. I'm skeptical. I think you are misunderstanding something. This is not for a pervasive ARC-only, statically guaranteed system. The best example he gives (and I agree with him) is iOS. Just look at the success of iOS, where the entire OS API is based on ARC (actually RC, with an option for both ARC and manual, but the latter is going away). If ARC was so bad, the iOS experience would show it. You may have doubts, but I can assure you I can build very robust and performant code with ARC in iOS. Let us consider that the single most difficult thing with memory management is giving away a piece of memory you created, never to see again, either via pushing into something, or returning it. The code that is responsible for creating that memory is not the code that is responsible for reclaiming it. This means you need an agreement between your code and the code that's accepting it, as to what the recipient needs to do once he's done. The giving away piece is easy, it's the when am I done with this? that makes it difficult. If it's going to one place, that's pretty simple, but if it's going to multiple places, that's where it becomes a headache. Basically, as long as someone is looking at this, it needs to exist, and you don't know who else is looking! In that respect, both ARC and GC handle the job admirably. But it so happens that some of the load that GC happens to require (more memory; infrequent but lagging pauses) is not conducive to the environments Manu is targeting (embedded games). The tradeoffs are not a complete win for ARC, and I don't think D should switch to ARC (I think it's impossible anyway), but providing a builtin compiler-assisted mechanism to use ARC would allow a style of coding that would complement D's existing tools. Consider that with D's version of pure, I can easily do functional and imperative programming, even mixing the two, without even thinking about it! I'd like to see something with ARC that makes it easy to intermix with GC, and replace some of the Object management in D. It can never be a fully generic solution, except by object wrapping, because the type needs to be aware of the RC. Other memory management tasks are simple. For example, owned arrays encapsulated inside an object, or temporary buffer space that you free before exiting a function. Those don't need complex memory management tools to make safe or effective. These have *defined lifetimes*. Compilers that do this do exist. I can't reconcile agreeing that ARC isn't good enough to be pervasive with compiler technology eliminates unnecessary ARC overhead. It's pretty pervasive on iOS. ARC has been around since iOS 4.3 (circa 2011). It's pretty difficult to use manual RC and beat ARC. In fact in some cases, ARC can beat manual, because the compiler has more insight and knowledge of the rules being followed. -Steve
Re: DIP60: @nogc attribute
On Monday, 21 April 2014 at 20:29:46 UTC, Steven Schveighoffer wrote: example he gives (and I agree with him) is iOS. Just look at the success of iOS, where the entire OS API is based on ARC (actually RC, with an option for both ARC and manual, but the latter is going away). If ARC was so bad, the iOS experience would show it. You may have doubts, but I can assure you I can build very robust and performant code with ARC in iOS. I am sure you are right about the last part, but the entire iOS API is not based on ARC. You only use Objective-C to obtain contexts and configure them. After that you do everything performance sensitive in pure C. That goes for everything from Quartz (which is C, not Objective-C), OpenGL (C, not Objective-C) to AudioUnits (C, not Objective-C). What is true is that you have bridges between manual ref counting where it exists in Core Foundation and Foundation, but just because you have a counter does not mean that you use it. ;-) It's pretty pervasive on iOS. ARC has been around since iOS 4.3 (circa 2011). Not if you are doing systems level programming. You are talking about application level programming and on that level it is pervasive, but iOS apps have advanced rendering-engines to rely on for audio/video.
Re: DIP60: @nogc attribute
On Monday, 21 April 2014 at 20:29:46 UTC, Steven Schveighoffer wrote: It's pretty difficult to use manual RC and beat ARC. In fact in some cases, ARC can beat manual, because the compiler has more insight and knowledge of the rules being followed. Are you sure? Have you tried to do it first with CFRelease/CFRetain, then with ARC? I believe this is the real reason (but I could be wrong): «You can’t implement custom retain or release methods.» from https://developer.apple.com/library/mac/releasenotes/ObjectiveC/RN-TransitioningToARC/Introduction/Introduction.html
Re: DIP60: @nogc attribute
Here are two very good reasons to avoid extensive ref-counting: 1. transactional memory ( you don't want a lock on two reads ) 2. cache coherency ( you don't want barriers everywhere ) Betting everything on ref counting is the same as saying no to upcoming CPUs. IMO that means ARC is DOA. It might be useful for some high level objects… but I don't understand why one would think that it is a good low level solution. It is a single-threaded solution.
Re: DIP60: @nogc attribute
On Sunday, 20 April 2014 at 18:48:49 UTC, John Colvin wrote: The way I understood your idea, was that a template could be marked @nogc, and yet still allow template arguments that themselves may gc. This can be accomplished by creating a unit test that passes non-allocating template parameters, and then verifying the instantiation is @nogc. The only way that works is if the unittest has coverage of all possible currently non-GC-using instantiations of all templates all the way down the call-tree.* Imagine the case where some function deep down the call-tree has a `static if(T == NastyType) doGCStuff();`. In order to protect against this, you have to check the internals of the entire call-tree in order to write the required unittest, and verify manually that you haven't missed a case every time anything changes. *alright, technically only those that can be instantiated by the function your testing, but this still blows up pretty fast. Looks like John has similar thinking pattern for this specific case :P Also you proposal does not add any hygiene checks to non-template functions _and_ requires to create a boilerplate output range mocks (for all possible duck types) and extra static asserts for all functions that you may want to mark as weak @nogc. I don't see it as clean solution that can actually be used in a library. The very marketing value of @nogc is not to show that something like it is possible (it already is) but to show hey, look how easy and clean it is!
Re: DIP60: @nogc attribute
On 4/21/2014 1:29 PM, Steven Schveighoffer wrote:
I think you are misunderstanding something. This is not for a pervasive
ARC-only, statically guaranteed system. The best example he gives (and I agree
with him) is iOS. Just look at the success of iOS, where the entire OS API is
based on ARC (actually RC, with an option for both ARC and manual, but the
latter is going away). If ARC was so bad, the iOS experience would show it.
You may have doubts, but I can assure you I can build very robust and performant
code with ARC in iOS.
The thing is, with iOS ARC, it cannot be statically guaranteed to be memory
safe. This makes it simply not acceptable for D in the general case. It works
with iOS because iOS allows all kinds of (unsafe) ways to escape it, and it must
offer those ways because it is not performant.
Kinda sorta memory safe, mostly memory safe, etc., is not a static guarantee.
There is JUST NO WAY that:
struct RefCount {
T* data;
int* count;
}
is going to be near as performant as:
T*
1. A dereference requires two indirections. Cache performance, poof!
2. A copy requires two indirections to inc, two indirections to dec, and an
exception unwind handler for dec.
3. Those two word structs add to memory consumption.
As you pointed out, performant code is going to have to cache the data* value.
That cannot be guaranteed memory safe.
I can't reconcile agreeing that ARC isn't good enough to be pervasive with
compiler technology eliminates unnecessary ARC overhead.
It's pretty pervasive on iOS. ARC has been around since iOS 4.3 (circa 2011).
Pervasive means for all pointers. This is not true of iOS. It's fine for iOS
to do a half job of it, because the language makes no pretensions about memory
safety. It is not fine for D to replace a guaranteed memory safe system with an
unsafe, hope-your-programmers-get-it-right, solution.
Re: DIP60: @nogc attribute
On Saturday, 19 April 2014 at 23:44:45 UTC, Marco Leise wrote:
Am Wed, 16 Apr 2014 20:32:20 +
schrieb Peter Alexander peter.alexander...@gmail.com:
On Wednesday, 16 April 2014 at 20:29:17 UTC, bearophile wrote:
Peter Alexander:
(I assume that nothrow isn't meant to be there?)
In D nothrow functions can throw errors.
Of course, ignore me :-)
You could do something like this:
void foo() @nogc
{
static err = new Error();
if (badthing)
{
err.setError(badthing happened);
throw err;
}
}
To be mutable err also needs to be __gshared.
But then it isn't thread safe. Two threads trying to set and
throw the same Error is a recipe for disaster.
Also: As far as I remember from disassembling C++, static
variables in functions are initialized on first access and
guarded by a bool. The first call to foo() would execute
err = new Error(); in that case. This code should not
compile under @nogc.
D static initialization doesn't work the same way. Everything is
initialized as the program is loaded, and everything must have a
statically known value. EG: An Error is allocated *somewhere*
(may or may not actually be the GC), and then the static value of
the Error is a pointer to that.
It's what allows us to do things like:
class A{}
struct S
{
A a = new A();
}
This is legal. All S.a will point the *same* A. S.init will be
initialized to point to that a.
Also, just doing this is good enough:
//
void foo() @nogc
{
static err = new Error(badthing happened);
if (badthing)
throw err;
}
//
It does require the message be known before hand, and not custom
built. But then again, where were you going to allocate the
message, and if allocated, who would clean it up?
That said, while the approach works, there could be issues with
re-entrance, or chaining exceptions.
Re: DIP60: @nogc attribute
On 2014-04-19 15:40, monarch_dodra wrote: Nonsense. It still works 99% of the time (I think only a subset of 100 letters in all of Unicode are affect, and even then, another 100 of them *shrink* on toUpper). It is really useful. It avoids *needles* allocations. Removing it would be more harmful than useful. I'm implicitly referring to toLowerInPlace as well. I'm pretty confident that most of the time it is used, people don't care *that* much that *absolutely* no allocation takes place. They just don't want to be wasteful. It still has a confusing name. Rename toUpperMaybeInPlace. Actually, the functionality is useful, it's just the name that is confusing. Then, for those that absolutely *can't* allocate provide a better interface. For example: `void toUpper(S, O)(S s, ref O o);` Burden on the caller to make it inplace from that (or to allocate accordingly when inplace is not possible). -- /Jacob Carlborg
Re: DIP60: @nogc attribute
On Wednesday, 16 April 2014 at 02:14:18 UTC, Walter Bright wrote: On 4/15/2014 6:57 PM, Mike wrote: I suspect some of the motivation for this is to give customers faster horses. I would be surprised if a @nogc attribute increased D's appeal, and I think efforts would be better allocated to some form of the above. Asking for @nogc comes up *constantly*. How bout this! Why not allow one to define their own attributes from a generalized subset and then define a few standard ones like @nogc. i.e., instead of having to define specific attributes every few years to satisfy some new thing, why not just abstract the process. Attributes, I believe, are essentially relationships between parts of code? If so, then one simply has to implement some generic way to specify the attributes and properties of the relationship to the compiler. Then anyone would have the tools to define and use these attributes as they wish. (in fact, I think it would just involve enhancing the current attribute support, probably just need to rewrite it all so that the same code is used for built in attributes(@safe, @pure, etc...) and user define attributes. So, we just need to define the attribute name and the properties it has such as: Assume Y uses X in some way(function call) and X has an attribute A defined on it: Inheritance - Y inherits attribute A. Exclusion - If Y has attribute B and B is mutually excluded from A then error Composition - If Y also uses Z and Z has attribute B then Y has the compound attribute (A:B). Compound attributes can be rewritten to other attributes using a grammar/reduction scheme. Some compositions can be invalid. E.g., @nogc and @gc, @pure and @notpure, etc... Duality - If an attribute A is not specified for a block of code then it's inverse attribute is implicitly specified always. e.g., @gc and @!gc = @nogc are duals and one or the other always is specified, even if implicit. etc... [Note, I'm not saying all attributes have these properties, just that these the possible properties they can have] By coming up with a general system(I'm sure there is some mathematical structure that describes attributes) it would be very easy to add attributes in the future and there would be a consistent code backing for them. It would also be easier for CT reflection on attributes. Anyways, just a thought, sounds easy in theory...
