Re: DIP1028 - Rationale for accepting as is
On Thursday, 28 May 2020 at 02:39:40 UTC, Jonathan M Davis wrote: On Wednesday, May 27, 2020 8:13:52 PM MDT Bruce Carneal via Digitalmars-d- announce wrote: On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis wrote: > [...] I remember reading a suggestion that additional linker symbols be emitted to carry the attribute and possibly type information while leaving the ABI untouched. Was this found to be impractical? Steven suggested something along those lines. I don't know how practical it would or wouldn't be, but I don't think that Walter even responded to the idea. Changing the DIP process to be 2 of 3 LMs required for acceptance could get us past all the "this is useless because Walter xyz..." road blocks. I've a proposal in to Mike for that now. We'll see how it goes. [snip] But regardless of whether DIP 1028 is the correct decision, the problem remains that your typical extern(C) function cannot be checked for @safety by the compiler, because it was compiled by a C compiler. The question then is just how the D compiler should treat them, and that's the main point of contention. There may be solutions like Steven suggested which would deal with edge cases where the implementation is in D but the linkage isn't, but ultimately, they're just edge cases. - Jonathan M Davis Yes. I understand the practical limits of machine checking and the 1028 issues. The edge case that I had in mind was piece wise replacement of C libraries with dlang reworkings and LCD FFI libraries written in dlang for Python and other languages. With the symbol additions, ABI could be decoupled from D-specific capabilities. Python and other C FFIs just work. Dlang programs get broader guarantees. Still, as you say, probably not a big concern.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, May 27, 2020 8:13:52 PM MDT Bruce Carneal via Digitalmars-d- announce wrote: > On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis wrote: > > On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via > > > > Digitalmars-d- announce wrote: > >> [...] > > > > Except that the linker matters a great deal in this discussion > > with regards to extern(D) functions, because @safe and @trusted > > are part of the name mangling for extern(D) functions. That > > means that if an extern(D) function declaration's attributes do > > not match its definition, then you'll get a linker error. So, > > treating non-extern(D) function declarations as @safe by > > default isn't necessarily a problem (though it would certainly > > work to just treat all function declarations as @system by > > default rather than treating extern(D) function declarations > > differently). The cases where non-extern(D) function > > declarations weren't actually @safe would be caught during the > > linking process. Sure, it would be nice if it were caught > > sooner, but you don't end up with them being invisibly treated > > @safe when they're not like we're going to get with DIP 1028. > > > > [...] > > I remember reading a suggestion that additional linker symbols be > emitted to carry the attribute and possibly type information > while leaving the ABI untouched. Was this found to be > impractical? Steven suggested something along those lines. I don't know how practical it would or wouldn't be, but I don't think that Walter even responded to the idea. Regardless, even if it worked perfectly, all such a solution would deal with would be the cases where you have a non-extern(D) function which is actually written in D and had its function body marked as @safe (and thus was verified to be @safe). Such functions definitely exist, but the main problem is really non-extern(D) functions which are actually written in C or C++ or whatnot. They won't have been checked for @safety. Right now, such declarations are treated by the compiler as @system by default, because it cannot check them for @safety, whereas with DIP 1028, it will then treat them as @safe in spite of the fact that it can't check them for @safety simply because Walter thinks that it would be too exceptional to treat them differently from the general rule of @safe by default and that if they're left as @system, too many people will just slap @trusted on their code as a whole to get the compiler to shut up. But regardless of whether DIP 1028 is the correct decision, the problem remains that your typical extern(C) function cannot be checked for @safety by the compiler, because it was compiled by a C compiler. The question then is just how the D compiler should treat them, and that's the main point of contention. There may be solutions like Steven suggested which would deal with edge cases where the implementation is in D but the linkage isn't, but ultimately, they're just edge cases. - Jonathan M Davis
Re: DIP1028 - Rationale for accepting as is
On Wednesday, May 27, 2020 6:24:25 PM MDT Andrei Alexandrescu via Digitalmars-
d-announce wrote:
> On 5/27/20 9:42 AM, Andrej Mitrovic wrote:
> > On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
> >> Un-annotated C declarations should be a red flag to any competent QA
> >> team. Recognizing a false @trusted is a whole lot harder.
> >
> > Is the actual problem those `@trusted:` declarations at the top of C
> > headers?
> >
> > There could be a simple solution to that:
> >
> > Ban `@trusted:` and `@trusted { }` which apply to multiple symbols. Only
>
> > allow `@trusted` to apply to a single symbol. For example:
>
> Oh wow what an interesting idea. Thanks.
I've argued for years that mass-applying any attribute is bad practice. It
makes it way too easy to accidentally apply an attribute to a function, and
experience has shown that it makes it _far_ easier to not realize that a
funtion already has a particular attribute (it's definitely happened in
Phobos PRs that an attribute has been mass-applied and then someone comes
along later and applies it to a specific function, because they didn't think
that that function had that attribute). Unfortunately, plenty of people seem
to love to mass apply attributes rather than marking each function
individually (presumably, because they hate having to use attributes all
over the place). And regardless of whether DIP 1028 is a good idea as-is,
the sad reality of the matter is that it's not uncommon even in druntime for
someone to slap @trusted: at the top of a module. It implies that the
declarations in question were not actually checked, and it's incredibly
error-prone when new declarations are added.
Personally, I normally only use : with access-level modifiers, and while I
like doing that (especially since I think that public and private functions
should normally be segregated anyway), I'm increasingly coming closer to the
conclusion that it's actually a bad idea in practice, because it's not
uncommon for people to not know which access-level modifier applies -
especially when looking at diffs in PRs.
I would _love_ to see it become illegal to mass-apply @trusted (or even
attributes in general), but I have no clue how easy it would be to get such
a DIP accepted or how much screaming there would be over it if it were
actually accepted.
- Jonathan M Davis
Re: DIP1028 - Rationale for accepting as is
On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis wrote: On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via Digitalmars-d- announce wrote: [...] Except that the linker matters a great deal in this discussion with regards to extern(D) functions, because @safe and @trusted are part of the name mangling for extern(D) functions. That means that if an extern(D) function declaration's attributes do not match its definition, then you'll get a linker error. So, treating non-extern(D) function declarations as @safe by default isn't necessarily a problem (though it would certainly work to just treat all function declarations as @system by default rather than treating extern(D) function declarations differently). The cases where non-extern(D) function declarations weren't actually @safe would be caught during the linking process. Sure, it would be nice if it were caught sooner, but you don't end up with them being invisibly treated @safe when they're not like we're going to get with DIP 1028. [...] I remember reading a suggestion that additional linker symbols be emitted to carry the attribute and possibly type information while leaving the ABI untouched. Was this found to be impractical?
Re: DIP1028 - Rationale for accepting as is
On Friday, May 22, 2020 9:29:03 AM MDT Timon Gehr via Digitalmars-d-announce wrote: > On 22.05.20 16:49, bachmeier wrote: > > I don't see that marking an extern(C) function @trusted buys you > > anything, at least not until you can provide a compiler guarantee for > > arbitrary C code. > > It buys you the ability to call that function from @safe code. Clearly > you can't mark it @safe because the compiler has not checked it. Yes, and @trusted in general buys you the ability to segregate and find code that is doing stuff that the compiler was unable to prove was @safe. Then when there's a bug related to memory safety, you know where to look. C functions cannot be @safe, because they haven't been vetted by the compiler. So, either what they're doing is actually memory safe (and it's fine to mark whem with @trusted), or what it's doing is memory safe if used correctly, and the calling code needs to be marked @trusted when using it correctly. Either way, the code with potential memory safety issues is segregated, and so when inevitably you do run into a memory safety bug, you know that it's either the @trusted code or the @system code that it calls which needs to be fixed. Of course, with this DIP, @safe no longer truly means that the code was vetted by @safety by the compiler but rather that all of the code that the compiler can see has been vetted for @safety, meaning that when there is an @safety bug, you must look not only for @trusted code, but you must look for non-extern(D) declarations which have been implictly treated as @trusted by the compiler. @safe still has value (and may even provide more value in that it will be used more often), but it provides much weaker guarantees in the process. - Jonathan M Davis
Re: DIP1028 - Rationale for accepting as is
On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via Digitalmars-d- announce wrote: > It kept being swapped about in the discussion thread, so I have been a > little on edge over people using non-extern(D). Because linkage doesn't > mean anything to anything but to cpu and linker in this discussion. Except that the linker matters a great deal in this discussion with regards to extern(D) functions, because @safe and @trusted are part of the name mangling for extern(D) functions. That means that if an extern(D) function declaration's attributes do not match its definition, then you'll get a linker error. So, treating non-extern(D) function declarations as @safe by default isn't necessarily a problem (though it would certainly work to just treat all function declarations as @system by default rather than treating extern(D) function declarations differently). The cases where non-extern(D) function declarations weren't actually @safe would be caught during the linking process. Sure, it would be nice if it were caught sooner, but you don't end up with them being invisibly treated @safe when they're not like we're going to get with DIP 1028. However, of course, with non-extern(D) declarations, even if the function definition is actually written in D, the fact that the function body was checked is not transmitted via the linking process, because it doesn't end up in the name mangling. So, the linkage used has a huge impact on whether you can rely on the @safe attribute on the function declaration actually meaning anything about whether the body was verified by the compiler. - Jonathan M Davis
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic wrote:
There could be a simple solution to that:
Ban `@trusted:` and `@trusted { }` which apply to multiple
symbols. Only allow `@trusted` to apply to a single symbol. For
example:
That came to my mind immediately. I'm not entirely sure about
`@trusted { }`, because it might be handy at times (I don't use
`@attribute { }` often). But `@trusted:` is such that it is
almost guaranteed that sooner or later it's applied to something
it wasn't intended to. I have no idea if that's the case with
braces.
Since `@trusted` is intended to be special, I'd go the full route
and ban it for any non-direct use. Mass-`@trusted` makes no sense
logically: If the programmer is required to investigate a
function to determine it's `@safe`-ty, the time required for the
investigation clearly outweighs the time marking the function
`@trusted` (if the investigation has that result, of course),
even if the judgment is fast.
If people go through a whole lot of `extern(C)` declarations and
marks them @trusted individually because neither `@trusted:` nor
`@trusted { }` did the job, it may occur to them it's probably
not the right thing.
I really hope this makes it through.
Re: DIP1028 - Rationale for accepting as is
On Friday, 22 May 2020 at 18:14:12 UTC, H. S. Teoh wrote: Then myFunc would become callable from @safe code, provided the passed-in argument is also @safe. The crucial point here is that while compiling myFunc, the compiler doesn't (need to) know the @safe-ty of `cb`, it can just treat it as an opaque object that it assumes the safety of, while it verifies the rest of the function body. This is parallel to C functions being of unverifiable safety, so if extern(C) functions were somehow marked and treated as opaque objects of unknown safety, then the compiler can still verify the rest of the code and produce a certificate of safety (modulo the C APIs used). If we had a way of expressing conditional safety, it could be a way to salvage @safe from this current situation. I had this thought a million times, tried to explain it at two DConfs but no-one (as it appeared to me) got the point. The main difference is, the function calling `myFunc` "knows" whether the delegate/function pointer it used as an argument is @safe or not. It's perfectly decidable. Personally, I'd even go that far and say, that's how all the function attributes (also pure, nothrow, @nogc) are intended to work.
Re: DIP1028 - Rationale for accepting as is
On 5/27/20 9:42 AM, Andrej Mitrovic wrote:
On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
Un-annotated C declarations should be a red flag to any competent QA
team. Recognizing a false @trusted is a whole lot harder.
Is the actual problem those `@trusted:` declarations at the top of C
headers?
There could be a simple solution to that:
Ban `@trusted:` and `@trusted { }` which apply to multiple symbols. Only
allow `@trusted` to apply to a single symbol. For example:
Oh wow what an interesting idea. Thanks.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 13:47:46 UTC, Claude wrote:
On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic
wrote:
Is the actual problem those `@trusted:` declarations at the
top of C headers?
There could be a simple solution to that:
Ban `@trusted:` and `@trusted { }` which apply to multiple
symbols. Only allow `@trusted` to apply to a single symbol. For
IMO, it makes things worse. Because the careless programmer
will slap @trusted to every declaration (maybe with a script or
find/replace macro of his editor). So now, we don't know if the
annotation is greenwashing or careful examination of the
definition.
At least with "@trusted:" and "@trusted { }", the greenwashing
is obvious.
the problem here ist: we are talking about careless programmers.
for those of us who actually care about safety it will be like:
1. ok lets write an extern(c) function.
2. forget to annotate it by accident.
3. hit compile
4. a) get an compiler error the hints that we missed that
annotation
or b) no error at all and we now have silently added unsafe code
to our codebase.
also, you can never trust that an @trusted function is actually
memory safe.
even if it was evaluated in the past. the addition of new code
(and even completely @safe code) can break any assumptions that
@trusted fun was relying on.
so a competent QA team _has_ to either check all @trusted code
fragments.
or with this dip check all @trusted and all extern functions.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 13:47:46 UTC, Claude wrote:
On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic
wrote:
Is the actual problem those `@trusted:` declarations at the
top of C headers?
There could be a simple solution to that:
Ban `@trusted:` and `@trusted { }` which apply to multiple
symbols. Only allow `@trusted` to apply to a single symbol. For
IMO, it makes things worse. Because the careless programmer
will slap @trusted to every declaration (maybe with a script or
find/replace macro of his editor). So now, we don't know if the
annotation is greenwashing or careful examination of the
definition.
The difference is when adding new symbols. If version control is
used, it would be very obvious in a review whether a new @trusted
symbol was added or not. A diff typically shows several lines of
context, and a toplevel `@trusted:` is easy to miss.
Re: DIP1028 - Rationale for accepting as is
On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
I have made these points before, but I'll summarize them here
for convenient referral.
[big snip of very long and arguably tangential Java screw-up and
other]
How does this relate to safe by default?
Consider the common (because that's how D started out) case of:
- clibrary.d
T massage_data(... many parameters ...);
... 200 more such declarations ...
- app.d --
import clibrary;
void useClibrary( ... parameters ...) {
massage_data(parameters);
}
-
This code, today, does not use annotations and it works. It's
been working
for a long time. Now, we implement the amendment, and it stops
compiling
because useClibrary is @safe and massage_data is @system. The
user is faced
with the following alternatives:
1. Go through 200 functions in clibrary.d and determine which
are @safe
and which are @system. This is what we want them to do. We try
to motivate
this with compiler error messages. Unfortunately, this is both
tedious and
thoroughly impractical, as our poor user Will Not Know which
are safe and
which are system. We can correctly annotate core.stdc.stdio
because I know
those functions intimately. This is not true for other system C
APIs, and
even less true for some third party C library we're trying to
interface to.
So, to condense and hopefully clarify: we can not use option 1)
and @system by default because "our poor user Will Not Know which
are safe and which are system"?
IOW, because someone unable to comment on safety would be
inconvenienced we enable them to inject more code into the
ecosphere by making @safe the default.
What am I missing?
2. Annotate useClibrary() as @trusted or @system. While easier,
this causes
all benefits to @safe by default to be lost.
3. Wrap the call to massage_data() with:
() @trusted { massage_data(parameters); } ();
If there are a lot of calls to clibrary, this is going to look
pretty awful.
Nobody likes writing or reading such ugly code. It's ok here
and there, but
not as a general thing.
4. Edit clibrary.d and make the first line:
@safe:
[snip regarding greenwashing and other]
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic wrote:
Is the actual problem those `@trusted:` declarations at the top
of C headers?
There could be a simple solution to that:
Ban `@trusted:` and `@trusted { }` which apply to multiple
symbols. Only allow `@trusted` to apply to a single symbol. For
IMO, it makes things worse. Because the careless programmer will
slap @trusted to every declaration (maybe with a script or
find/replace macro of his editor). So now, we don't know if the
annotation is greenwashing or careful examination of the
definition.
At least with "@trusted:" and "@trusted { }", the greenwashing is
obvious.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 10:07:48 UTC, Johannes Loher wrote: Am 27.05.20 um 11:50 schrieb Walter Bright: [...] Un-annotated C declarations should be a red flag to any competent QA team. Recognizing a false @trusted is a whole lot harder. [...] Also in my opinion, a competent QA department should carefully look at any @trusted code /declarations. Maybe it is not a "red flag" but it is definitely something that needs to be checked with extra care. I think additionally we shouldn't make D code review harder to do than it needs to be. For a reviewer "something correct which is missing" is harder to pick up than "something wrong which is there" (@trusted/@system attributes) When I, as an incompetent but trying QA department, come across a code change introducing a new module wrapping a C library and all tests pass, I would most likely not notice that the author did not type `@system:`. If there was an `@trusted:` at the top of the file it would be picked up by D-Scanner and help in code review. And well if someone slapped `@safe:` at the top of the file... read on: Someone in this thread wrote something like "@safe" should be forbidden on functions without method body and instead only use @trusted or @system. I think it might even be you who brought it up? I love the idea of forbidding @safe when the compiler can't actually check the content of a function. @safe, @trusted and @system could then have perfect definitions without edge cases. And you would change the mangling of @safe and @trusted to be the same which was also brought up in this thread.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
Un-annotated C declarations should be a red flag to any
competent QA team. Recognizing a false @trusted is a whole lot
harder.
Is the actual problem those `@trusted:` declarations at the top
of C headers?
There could be a simple solution to that:
Ban `@trusted:` and `@trusted { }` which apply to multiple
symbols. Only allow `@trusted` to apply to a single symbol. For
example:
---
@trusted:
extern(C) void memcpy (void*, void*, size_t);
extern(C) void write42 (void*);
---
Error.
---
@trusted extern(C) void memcpy (void*, void*, size_t);
@trusted extern(C) void write42 (void*);
---
OK, compiles. The bindings author has clearly added @trusted
manually to each symbol.
Obviously there are escape hatches like mixins, code generation,
etc.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 11:46:40 UTC, Andrei Alexandrescu wrote: On 5/27/20 6:22 AM, Walter Bright wrote: On 5/27/2020 3:06 AM, rikki cattermole wrote: Open to everybody and it can be recorded by Twitch. I've done the video conferencing many times over the decades. I just don't find it to be productive. Maybe it's some defect in me. They usually go something like this: [attempt at joke snipped] It's safe to assume things have improved over time. Currently because COVID-19 we have (At least where I live) Politicians having meetings online and open to the public and it goes without any problems. I've been saying this since last year, why not go for online meetings to fix things instead of only using this Forum which sometimes posts get neglected and ideas misunderstood. Ouch this is about Technology and people don't use it the right way. Matheus.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 10:03:21 UTC, Walter Bright wrote: It's not an acceptable solution for people who require correctness. ZombineDev can't tell an auditor that he relies on D-Scanner to find the problematic spots. I believe it's unavoidable to provide an option to change the default. It could make extern @system or refuse to compile safe code calling @safe extern. People who require correctness are going to have to manually audit each and every C declaration regardless. There's no way around it. A less strict auditor would focus his attention on un-annotated declarations and assume that annotated ones are annotated correctly. A less strict auditor wouldn't have to look at un-annotated extern(C) functions if they were @system by default. It's more work for them to ensure they are actually @safe. The majority will be un-annotated if you leave it as @safe.
Re: DIP1028 - Rationale for accepting as is
On 27/5/20 11:50, Walter Bright wrote: It is a fair point. But I am looking a bit farther than that - the team that is responsible for QAing software (sometimes it is a separate team). The QA team cannot tell the difference between correctly annotated @trusted code and greenwashed code. This is a most unsubstantiated assertion. If this were the case, I would have to wonder what the QA team is there for, if they can just be replaced by an automatic lint tool. *AND* even if that were so, see below: But they can tell when code is not annotated (no, it is not harder to do, the annotations are designed to not be hidden - annotations cannot be hidden by the preprocessor nor are they propagated from imports. They have to be there, and if grep doesn't find them, they are not there. I've never had any difficulty finding the annotations belonging to a declaration). Then the coder just slaps `@safe:`, `@trusted:`, or whatever greenwashing you were trying to avoid, and we're just back to square one since, per your previous assertion, QA can't notice the difference. Heck, in such an environment I might slap `@trusted:` just in case at the beginning of each and every file and live in a land of bliss thereafter! Un-annotated C declarations should be a red flag to any competent QA team. Recognizing a false @trusted is a whole lot harder. Exactly, which is why they should be disallowed by the compiler. Considering them `@system` by default is a way to do it. Another (perhaps even better) already proposed one would be to actually consider them `@safe`, but disallow `@safe` declarations where the code can't be analyzed by the compiler (it would be debatable if the mangling should be enough or not). BTW, one good thing that has come out of this issue is people are strongly in favor of what @safe does. That bodes well for DIP1000 and the @live code, for a long time I seemed to be the only one who cared about it. Being in favour of what @safe purports to do and in favour of what it actually does are two very different things. The whole selling point of @safe used to be "you're only responsible for `@trusted` code, now that's no longer true (or at least even less, if we account for the already existing holes). The direction of travel is opposite to the announced destination. I also have no problem if the D style checker checked for un-annotated C declarations. That's the kind of thing it's for. So, if you leave them for the style checker that means that you envision cases where this could be the desired behaviour, otherwise you should just let the compiler error. What would these cases look like? Because I can't imagine any.
Re: DIP1028 - Rationale for accepting as is
On 5/27/20 6:22 AM, Walter Bright wrote: On 5/27/2020 3:06 AM, rikki cattermole wrote: Open to everybody and it can be recorded by Twitch. I've done the video conferencing many times over the decades. I just don't find it to be productive. Maybe it's some defect in me. They usually go something like this: [attempt at joke snipped] It's safe to assume things have improved over time.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 10:40:18 UTC, Walter Bright wrote: On 5/27/2020 3:07 AM, Johannes Loher wrote: This is a very specific situation. There are a lot of teams / developers that do not work in this manner. I don't know the numbers so I will make no statement about what is more common but my personal experience is a different one. I've seen larger companies operate this way. Smaller ones cannot afford to. Also what is the difference between your QA department telling you to correctly annotate C declarations and the compiler telling you that? > If you expect people to ignore what the compiler is telling them, why do > you expect them to listen to the QA department? The QA dept is motivated to not be taken in by greenwashing. (Back in my days at Boeing, I worked in the design department. There was an entirely separate organization call the Stress Group. Their job was to sign off on every nitpicky detail. If they signed off on something that turned out to be wrong, it was bad for their careers. Your designs did not move forward without Stress signoff. If you tried to trick Stress, that was the end of your career at Boeing.) So yeah, there's a huge difference between tricking the compiler and tricking the QA department. In my opinion, the compiler actually _is_ one of the best QA departments. Indeed it is, and that's the whole point to @safe. My motivation here is make suspicious code stand out. @trusted code does not stand out so much, because it is required to exist. I'm utterly confused by this. Trusted code is always suspicious and always stands out because it is not formally verified whereas @safe code should be bulletproof. Why would anyone bother to manually check code that is supposed to be @safe? Trusted code is the villain here; wherever you see @trusted you know that the code might contain vulnerabilities because it was audited by a human and not by a machine. If you have @safe code that has memory safety issues (from callinc C functions) then what's the point of @safe? Also in my opinion, a competent QA department should carefully look at any @trusted code /declarations. Maybe it is not a "red flag" but it is definitely something that needs to be checked with extra care. Looking at un-annotated declarations should be the first step. If it is annotated with @trusted, at least there's the coffee stain on the drawing indicating that somebody looked at it.
Re: DIP1028 - Rationale for accepting as is
On 5/27/20 6:03 AM, Walter Bright wrote: On 5/26/2020 5:20 AM, Johannes T wrote: On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: [..] Thank you very much for your patience with all the negative feedback. I get your decision to not annotate extern C with @system by default. The biggest issue with extern @system is that @trusted would become less useful when dealing with declarations and bindings. @trusted would appear more frequently. We wouldn't be able to assume that the author put in effort in assessing the trustworthiness of a declaration. More greenwashing, less safety. Those are severe drawbacks, I agree. However, as Andrei pointed out, PR is a huge problem. We need to be able to sell it with a straight face. Frankly, I feel that if I could sit down with you folks, I can get the idea across what I'm trying to accomplish. Didn't work with me.
Re: DIP1028 - Rationale for accepting as is
Am 27.05.20 um 12:40 schrieb Walter Bright: > > Indeed it is, and that's the whole point to @safe. My motivation here is > make suspicious code stand out. @trusted code does not stand out so > much, because it is required to exist. By that logic, wouldn't it also make more sense to implicitly default to @trusted instead of @safe for any unannotated D code? Following that logic, if @safe is the default and somebody writes non-@safe code without annotating it, it won't compile they might just slap @trusted on it without actually checking it. By that they make it harder for the QA department to spot. If @trusted was the default, it would compile, there would be no annotation on it an the QA department would see it is a red flag. What's the difference here?
Re: DIP1028 - Rationale for accepting as is
On 5/27/2020 3:07 AM, Johannes Loher wrote: This is a very specific situation. There are a lot of teams / developers that do not work in this manner. I don't know the numbers so I will make no statement about what is more common but my personal experience is a different one. I've seen larger companies operate this way. Smaller ones cannot afford to. Also what is the difference between your QA department telling you to correctly annotate C declarations and the compiler telling you that? > If you expect people to ignore what the compiler is telling them, why do > you expect them to listen to the QA department? The QA dept is motivated to not be taken in by greenwashing. (Back in my days at Boeing, I worked in the design department. There was an entirely separate organization call the Stress Group. Their job was to sign off on every nitpicky detail. If they signed off on something that turned out to be wrong, it was bad for their careers. Your designs did not move forward without Stress signoff. If you tried to trick Stress, that was the end of your career at Boeing.) So yeah, there's a huge difference between tricking the compiler and tricking the QA department. In my opinion, the compiler actually _is_ one of the best QA departments. Indeed it is, and that's the whole point to @safe. My motivation here is make suspicious code stand out. @trusted code does not stand out so much, because it is required to exist. Also in my opinion, a competent QA department should carefully look at any @trusted code /declarations. Maybe it is not a "red flag" but it is definitely something that needs to be checked with extra care. Looking at un-annotated declarations should be the first step. If it is annotated with @trusted, at least there's the coffee stain on the drawing indicating that somebody looked at it.
Re: DIP1028 - Rationale for accepting as is
Am 27.05.20 um 11:50 schrieb Walter Bright: > > It is a fair point. > By the way, thank you for acknowledging that. I appreciate it.
Re: DIP1028 - Rationale for accepting as is
On 27/05/2020 10:12 PM, Johannes Loher wrote: Am 27.05.20 um 11:25 schrieb Walter Bright: On 5/24/2020 3:40 AM, Stefan Koch wrote: The distinction is that you can find a slapped on trusted with a grep. It's just as easy to use grep to *not* find @trusted. But that's not enough. You need a regexp that searches for extern (C(++)) declarations that do not have any of @safe, @trusted, @system. The attributes can also be either before the return type + name + parameters or after it. They can also be mixed with any other attributes. Sure, you can probably write a regex that matches all of this but it is a _lot_ more complicated than simply searching for @trusted. extern(Windows) extern(System) COM Few more things there.
Re: DIP1028 - Rationale for accepting as is
On 5/27/2020 3:06 AM, rikki cattermole wrote: Open to everybody and it can be recorded by Twitch. I've done the video conferencing many times over the decades. I just don't find it to be productive. Maybe it's some defect in me. They usually go something like this: "Who else is here?" "Hello ... Hello?" "Can anybody hear me? Is my mike working?" "Where's Bob? Is he connected?" "I just got my connection working again. What did I miss?" "Bob? Bob?" "Can someone go find Bob?" "Can you say that again? The audio cut out." "Bob here!" "Hmm. We lost Bob again?" "Fred, can you please turn off your motorcycle?" "Bob here again! Sorry!" "Fred? Now where's Fred? Fred?" What has always worked best for me is a face to face over coffee and/or beer. Which is why I'm missing DConf this year terribly.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 10:06:41 UTC, rikki cattermole wrote: On 27/05/2020 10:03 PM, Walter Bright wrote: Frankly, I feel that if I could sit down with you folks, I can get the idea across what I'm trying to accomplish. Okay, how is your camera and mic situation? Lets do a Twitch stream, make sure there are some moderators in place as well. Open to everybody and it can be recorded by Twitch. Youtube streams work better in my experience.
Re: DIP1028 - Rationale for accepting as is
Am 27.05.20 um 11:25 schrieb Walter Bright: > On 5/24/2020 3:40 AM, Stefan Koch wrote: >> The distinction is that you can find a slapped on trusted with a grep. > > It's just as easy to use grep to *not* find @trusted. But that's not enough. You need a regexp that searches for extern (C(++)) declarations that do not have any of @safe, @trusted, @system. The attributes can also be either before the return type + name + parameters or after it. They can also be mixed with any other attributes. Sure, you can probably write a regex that matches all of this but it is a _lot_ more complicated than simply searching for @trusted.
Re: DIP1028 - Rationale for accepting as is
On 5/26/2020 6:07 AM, Panke wrote: On Tuesday, 26 May 2020 at 12:20:31 UTC, Johannes T wrote: On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: [..] Thank you very much for your patience with all the negative feedback. Yes, good think to stop once in a while and appreciate it. "To not complain is sufficient praise", does not always cut it. Thank you, it is nice to see this.
Re: DIP1028 - Rationale for accepting as is
Am 27.05.20 um 11:50 schrieb Walter Bright: > It is a fair point. But I am looking a bit farther than that - the team > that is responsible for QAing software (sometimes it is a separate > team). The QA team cannot tell the difference between correctly > annotated @trusted code and greenwashed code. > > But they can tell when code is not annotated (no, it is not harder to > do, the annotations are designed to not be hidden - annotations cannot > be hidden by the preprocessor nor are they propagated from imports. They > have to be there, and if grep doesn't find them, they are not there. > I've never had any difficulty finding the annotations belonging to a > declaration). > > Un-annotated C declarations should be a red flag to any competent QA > team. Recognizing a false @trusted is a whole lot harder. This is a very specific situation. There are a lot of teams / developers that do not work in this manner. I don't know the numbers so I will make no statement about what is more common but my personal experience is a different one. Also what is the difference between your QA department telling you to correctly annotate C declarations and the compiler telling you that? If you expect people to ignore what the compiler is telling them, why do you expect them to listen to the QA department? In my opinion, the compiler actually _is_ one of the best QA departments. Also in my opinion, a competent QA department should carefully look at any @trusted code /declarations. Maybe it is not a "red flag" but it is definitely something that needs to be checked with extra care.
Re: DIP1028 - Rationale for accepting as is
On 27/05/2020 10:03 PM, Walter Bright wrote: Frankly, I feel that if I could sit down with you folks, I can get the idea across what I'm trying to accomplish. Okay, how is your camera and mic situation? Lets do a Twitch stream, make sure there are some moderators in place as well. Open to everybody and it can be recorded by Twitch.
Re: DIP1028 - Rationale for accepting as is
On 5/26/2020 5:20 AM, Johannes T wrote: On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: [..] Thank you very much for your patience with all the negative feedback. I get your decision to not annotate extern C with @system by default. The biggest issue with extern @system is that @trusted would become less useful when dealing with declarations and bindings. @trusted would appear more frequently. We wouldn't be able to assume that the author put in effort in assessing the trustworthiness of a declaration. More greenwashing, less safety. Those are severe drawbacks, I agree. However, as Andrei pointed out, PR is a huge problem. We need to be able to sell it with a straight face. Frankly, I feel that if I could sit down with you folks, I can get the idea across what I'm trying to accomplish. I think this might be done by formally redefining the meaning of safety attributes when applied to extern: The safety of extern can't be assessed by the compiler. Implicit "@safe" extern within @safe code is accepted for the convenience of the average user. Explicit @system or @trusted can be added for desired behavior. I think this can be explained. I do, too. But apparently not by typing. It's not an acceptable solution for people who require correctness. ZombineDev can't tell an auditor that he relies on D-Scanner to find the problematic spots. I believe it's unavoidable to provide an option to change the default. It could make extern @system or refuse to compile safe code calling @safe extern. People who require correctness are going to have to manually audit each and every C declaration regardless. There's no way around it. A less strict auditor would focus his attention on un-annotated declarations and assume that annotated ones are annotated correctly.
Re: DIP1028 - Rationale for accepting as is
On 27/05/2020 9:50 PM, Walter Bright wrote: BTW, one good thing that has come out of this issue is people are strongly in favor of what @safe does. That bodes well for DIP1000 and the @live code, for a long time I seemed to be the only one who cared about it. Most of the arguments against @safe by default I have seen over the years (including from me) originate in /how/ to make it default, not in it becoming the default. For me at least, I will not ever want to use @live, pointers with multiple behaviors depending on how it is used which don't change the syntax? No thanks, I thought we had learned that was a bad idea. But a head const storage class that handles lifetimes with no extra syntax thanks to DIP25/1000, yes please! I will use that.
Re: DIP1028 - Rationale for accepting as is
On 5/26/2020 12:18 AM, Johannes Loher wrote: Just think about what the developer's reaction would be in the situation I described in my last post, when he actually finds the issue. In your variant, the developer will be questioning why the compiler did not help him at all with realizing that there might be a problem. It is very likely that he blames the issue on the compiler. In the other variant, he will realize that this something he had been warned about by the compiler and that he made a careless wrong decision and that it is his own fault. In the future he will be likely more careful when it comes to annotating stuff with @trusted. So this even has an educational effect. It is a fair point. But I am looking a bit farther than that - the team that is responsible for QAing software (sometimes it is a separate team). The QA team cannot tell the difference between correctly annotated @trusted code and greenwashed code. But they can tell when code is not annotated (no, it is not harder to do, the annotations are designed to not be hidden - annotations cannot be hidden by the preprocessor nor are they propagated from imports. They have to be there, and if grep doesn't find them, they are not there. I've never had any difficulty finding the annotations belonging to a declaration). Un-annotated C declarations should be a red flag to any competent QA team. Recognizing a false @trusted is a whole lot harder. BTW, one good thing that has come out of this issue is people are strongly in favor of what @safe does. That bodes well for DIP1000 and the @live code, for a long time I seemed to be the only one who cared about it. I also have no problem if the D style checker checked for un-annotated C declarations. That's the kind of thing it's for.
Re: DIP1028 - Rationale for accepting as is
On 5/24/2020 3:40 AM, Stefan Koch wrote: The distinction is that you can find a slapped on trusted with a grep. It's just as easy to use grep to *not* find @trusted.
Re: DIP1028 - Rationale for accepting as is
On Wednesday, 27 May 2020 at 02:42:24 UTC, Walter Bright wrote: On 5/24/2020 6:04 PM, Timon Gehr wrote: Implicit greenwashing by the compiler is a nuisance that makes it harder to do the job correctly and easier to do the wrong thing. You and I are just going to disagree about that. Unfortunately science and research Doesn't agree with you. https://en.m.wikipedia.org/wiki/Default_effect
Re: DIP1028 - Rationale for accepting as is
On 5/24/2020 6:04 PM, Timon Gehr wrote: Implicit greenwashing by the compiler is a nuisance that makes it harder to do the job correctly and easier to do the wrong thing. You and I are just going to disagree about that.
Re: DIP1028 - Rationale for accepting as is
On 5/25/20 5:17 AM, Walter Bright wrote: On 5/23/2020 5:13 AM, Steven Schveighoffer wrote: And let's be honest here, if you are OK with it, putting @trusted: at the top of your extern(C) functions is fine with me. At least that's not a lie. @trusted says the interface to the function is safe. If the programmer did not actually check the interface, it is greenwashing. For example, memcpy() does not have a safe interface and marking it @trusted doesn't improve matters. Yes, and the programmer did this on purpose. To quote someone else `My reasoning at the time was "I trust libclang".` In that sense, it's not a lie, or "incorrect," you are just trusting that whomever wrote that C library (maybe it was the author of the prototype) followed strictly the D safety rules. It's possible to write a function that takes 2 pointers and a size_t that is @safe, just like memcpy (it won't do anything, but there is nothing inherently wrong with that). If it were possible to determine directly from the types of parameters that a function wasn't safe, we would not be having this discussion. So "trusting" a library regardless of interface is what @trusted: does. And while I wouldn't do that, or use a library that does that, at least I can see where the problem is! @trusted code is hard to write correctly, and there's going to be greenwashing abound when this change comes about. I can even envision someone *recommending* "Don't import core.stdc.x, just declare the prototype yourself, and you can then call them from safe functions!". Greenwashing by the compiler essentially adding @trusted to every C porototype doesn't make it any better. I'd much rather have a clear indication that someone made this conscious decision, correct or not, and here is where you should look to find the problems. If I am interested in ENSURING that all code I use is @safe or correctly @trusted, then I know how to find those points of contention if the @trusted attribute is required where it should be. The biggest problem with the magic greenwashing of this DIP is that there is 10-20 years of existing D code with extern(C) function prototypes that are specifically unmarked BECAUSE the author knew that meant @system. I don't know how this DIP can be approved just on that problem alone. I hope something better comes out of this discussion, I'm going to mute it now because I don't have the time to keep arguing against people who don't seem to understand either memory safety or human nature. -Steve
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 12:51:59 UTC, Atila Neves wrote: On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote: On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov [ZombineDev] wrote: [...] If the greenwashing part was separated and delayed it would give time to find out if Walters hypothesis about people just doing it themselves is true. -preview=safe now -revert=safe "tomorrow" If these are permanent, an ugly fork will have occurred in all but name. If these are temporary, we'll have gained little since the damage caused by assuming human checked routines are @safe will remain after the switches are removed.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 12:51:59 UTC, Atila Neves wrote: On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote: On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov [ZombineDev] wrote: [...] If the greenwashing part was separated and delayed it would give time to find out if Walters hypothesis about people just doing it themselves is true. -preview=safe now -revert=safe "tomorrow" Does that separate "safe by default" from "c functions are assumed safe"?
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 13:21:08 UTC, Johannes Loher wrote: Am 26.05.20 um 15:10 schrieb Panke: The bazel community has lots of such switches. Basically every new behaviour get's introduced with a --preview switch, that will turn into a --revert after some time. For each switch there is a github issue, explaining the change it detail. Why it was necessary, what the changed behaviour is, how to migrate and the timeline for this particular switch. D has the same thing (e.g. -preview=safedefault will enable this DIP). The main difference to Johannes T's suggestion is that these preview and revert switches are only a temporary measure to ease the transition. They do not create different versions of the language permanently. What we don't have is good documentation and a clear timeline for the switches.
Re: DIP1028 - Rationale for accepting as is
Am 26.05.20 um 15:10 schrieb Panke: > > The bazel community has lots of such switches. Basically every new > behaviour get's introduced with a --preview switch, that will turn into > a --revert after some time. > > For each switch there is a github issue, explaining the change it > detail. Why it was necessary, what the changed behaviour is, how to > migrate and the timeline for this particular switch. D has the same thing (e.g. -preview=safedefault will enable this DIP). The main difference to Johannes T's suggestion is that these preview and revert switches are only a temporary measure to ease the transition. They do not create different versions of the language permanently.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 12:38:54 UTC, Johannes Loher wrote: A switch might help but the whole purpose of this DIP is to get the defaults right. In my opinion, such a switch should be enabled by default. There is also the danger of creating 2 versions of the language that are incompatible with each other. Only giving this a quick thought, I believe it could work (you can still link libraries compiled with and without the switch) but I'm not completely sure. The bazel community has lots of such switches. Basically every new behaviour get's introduced with a --preview switch, that will turn into a --revert after some time. For each switch there is a github issue, explaining the change it detail. Why it was necessary, what the changed behaviour is, how to migrate and the timeline for this particular switch.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 12:20:31 UTC, Johannes T wrote: On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: [..] Thank you very much for your patience with all the negative feedback. Yes, good think to stop once in a while and appreciate it. "To not complain is sufficient praise", does not always cut it.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote: On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov [ZombineDev] wrote: [...] If the greenwashing part was separated and delayed it would give time to find out if Walters hypothesis about people just doing it themselves is true. -preview=safe now -revert=safe "tomorrow"
Re: DIP1028 - Rationale for accepting as is
Am 26.05.20 um 14:20 schrieb Johannes T: > Thank you very much for your patience with all the negative feedback. I > get your decision to not annotate extern C with @system by default. As much as i disagree with the decision, I am still very glad that we are at least having a discussion. I also want to thank Walter for participating here. I understand this can be difficult but it is the right thing to do. > The biggest issue with extern @system is that @trusted would become less > useful when dealing with declarations and bindings. @trusted would > appear more frequently. We wouldn't be able to assume that the author > put in effort in assessing the trustworthiness of a declaration. More > greenwashing, less safety. Those are severe drawbacks, I agree. > However, as Andrei pointed out, PR is a huge problem. We need to be able > to sell it with a straight face. > I think this might be done by formally redefining the meaning of safety > attributes when applied to extern: > The safety of extern can't be assessed by the compiler. Implicit "@safe" > extern within @safe code is accepted for the convenience of the average > user. Explicit @system or @trusted can be added for desired behavior. > I think this can be explained. Well, the spec actually already mentions that: > Safe External Functions > > External functions don't have a function body visible to the compiler: > > @safe extern (C) void play(); > > and so safety cannot be verified automatically. > Best Practices: Explicitly set an attribute for external functions rather than relying on default settings. I still don't think this is the correct thing to do. Instead, annotating external (at least extern(C)) functions with @safe (implicitly as the default or explicitly) should be a compile error in my opinion. > It's not an acceptable solution for people who require correctness. > ZombineDev can't tell an auditor that he relies on D-Scanner to find the > problematic spots. I believe it's unavoidable to provide an option to > change the default. It could make extern @system or refuse to compile > safe code calling @safe extern. A switch might help but the whole purpose of this DIP is to get the defaults right. In my opinion, such a switch should be enabled by default. There is also the danger of creating 2 versions of the language that are incompatible with each other. Only giving this a quick thought, I believe it could work (you can still link libraries compiled with and without the switch) but I'm not completely sure.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov [ZombineDev]
wrote:
On Monday, 25 May 2020 at 23:39:33 UTC, Andrei Alexandrescu
wrote:
[..]
Thank you, Andrei, you've put this quite eloquently. With more
than 200+ replies, unfortunately, this whole discussion looks
like an excessively inefficient use of the community's time.
One way to resolve this stalemate (which I've proposed in
another post) is to split off the point of contention from
DIP1028. In other words, amend the newly proposed compiler
switch `-preview=safedefault` to not change the meaning of
non-extern(D) function declarations and introduce a new
compiler switch `-implicitly-{safe,trusted}-extern-fn-decls`
(*) that adds the automatic "greenwashing" functionality that
Walter desires so much.
If the greenwashing part was separated and delayed it would give
time to find out if Walters hypothesis about people just doing it
themselves is true.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: [..] Thank you very much for your patience with all the negative feedback. I get your decision to not annotate extern C with @system by default. The biggest issue with extern @system is that @trusted would become less useful when dealing with declarations and bindings. @trusted would appear more frequently. We wouldn't be able to assume that the author put in effort in assessing the trustworthiness of a declaration. More greenwashing, less safety. Those are severe drawbacks, I agree. However, as Andrei pointed out, PR is a huge problem. We need to be able to sell it with a straight face. I think this might be done by formally redefining the meaning of safety attributes when applied to extern: The safety of extern can't be assessed by the compiler. Implicit "@safe" extern within @safe code is accepted for the convenience of the average user. Explicit @system or @trusted can be added for desired behavior. I think this can be explained. It's not an acceptable solution for people who require correctness. ZombineDev can't tell an auditor that he relies on D-Scanner to find the problematic spots. I believe it's unavoidable to provide an option to change the default. It could make extern @system or refuse to compile safe code calling @safe extern.
Re: DIP1028 - Rationale for accepting as is
On 26.05.20 13:09, Atila Neves wrote: On Monday, 25 May 2020 at 17:01:24 UTC, Panke wrote: On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote: A few years ago I submitted several PRs to Phobos to mark all unittests that could with @safe explicitly. I'd say that was a good example of nobody reviewing them for their @systemness. Ideally you should be able to blindly mark every function definition with @safe, because the compiler will catch you if you fall. Only if you type @trusted you should need to be careful. Doesn't work for templated functions since their @safety might depend on the the particular instantiation (consider std.algorithm.map). I think the point was that annotating with @safe liberally should not enable memory corruption.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 17:01:24 UTC, Panke wrote: On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote: A few years ago I submitted several PRs to Phobos to mark all unittests that could with @safe explicitly. I'd say that was a good example of nobody reviewing them for their @systemness. Ideally you should be able to blindly mark every function definition with @safe, because the compiler will catch you if you fall. Only if you type @trusted you should need to be careful. Doesn't work for templated functions since their @safety might depend on the the particular instantiation (consider std.algorithm.map).
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: On 5/25/2020 7:04 PM, Johannes Loher wrote: Now let's compare the two different options: 1. With DIP1028 in its current form, the code will compile and a memory corruption will actually happen. The problem might be extremely difficult to track down for the developer because he has no clues whatsoever where to start looking. 2. With one of the solutions that were presented, the code will not compile as it is. According to your argument of „convenience“, the developer will probably just mark the function incorrectly as @trusted which makes the code compile. The memory corruption will happen. However, even if the developer did not think much about potential safety issues when adding @trusted to the function, he now still remembers that he did that (it was a conscious decision, even if it was a careless and lazy one). He has a clear point to start looking for the reason of the memory corruption. Do you honestly think option 1 is better? Yes, for reasons I carefully laid out. > no clues whatsoever He can look at unattributed declarations. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts. I think this is what's the most frustrating part. You aren't actually taking criticism if you cherry-pick what you reply to.
Re: DIP1028 - Rationale for accepting as is
On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote: 2. Make @safe the default for _all_ declarations but still make @safe on declarations without code a compile error (because the compiler cannot verify it). This means that annotating function declarations without code with @system or @trusted is now mandatory. This is what Timon has been arguing for if I understood him correctly. If this DIP is going to be implemented "as is", this should be added in the future. It could also be a warning, but that is not something that D often does. With this addition, not having a special case for extern(), is a positive.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: He can look at unattributed declarations. The issue is, you have to know about that beforehand. In the other situation, you explicitly get warned by the compiler (by the compile error) that that something dangerous is going on. Also, this means, there are now 2 different things to look for: @trusted function declarations and unannotated extern(C) declarations. The latter is much harder to search for (as Andrei also noted in his reply) and it's just much simpler to only have one escape hatch to look out for. It is also a very weird special case. @safe means no memory corruption is possible except for any @trusted declarations and any unannotated extern(C) declarations, wrongfully @safe extern(C) declarations and wrongfully @trusted extern(C) declarations. This just doesn’t sound right. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts. The question should be rephrased: „Is explicit greenwashing by the developer better, more honest, more debuggable than implicit greenwashing by the compiler?“ Yes for better and more debuggable for the reasons that have already been mentioned so many times. Regarding honesty: In your variant, the compiler is dishonest, in the other variant, the developer is dishonest. Just think about what the developer's reaction would be in the situation I described in my last post, when he actually finds the issue. In your variant, the developer will be questioning why the compiler did not help him at all with realizing that there might be a problem. It is very likely that he blames the issue on the compiler. In the other variant, he will realize that this something he had been warned about by the compiler and that he made a careless wrong decision and that it is his own fault. In the future he will be likely more careful when it comes to annotating stuff with @trusted. So this even has an educational effect.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 02:04:02 UTC, Johannes Loher wrote: According to your argument of „convenience“, the developer will probably just mark the function incorrectly as @trusted which makes the code compile. The memory corruption will happen. However, even if the developer did not think much about potential safety issues when adding @trusted to the function, he now still remembers that he did that (it was a conscious decision, even if it was a careless and lazy one). He has a clear point to start looking for the reason of the memory corruption. Or he'll do the right thing, and keep the function @system but call it from a @trusted block. Then when he's stuck debugging, he can show it to his experienced buddy and his experienced buddy will instantly look in the @trusted block because it's suspicious.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 23:39:33 UTC, Andrei Alexandrescu wrote:
[..]
Thank you, Andrei, you've put this quite eloquently. With more
than 200+ replies, unfortunately, this whole discussion looks
like an excessively inefficient use of the community's time.
One way to resolve this stalemate (which I've proposed in another
post) is to split off the point of contention from DIP1028. In
other words, amend the newly proposed compiler switch
`-preview=safedefault` to not change the meaning of non-extern(D)
function declarations and introduce a new compiler switch
`-implicitly-{safe,trusted}-extern-fn-decls` (*) that adds the
automatic "greenwashing" functionality that Walter desires so
much.
(*) I think that `-implicitly-safe-extern-fn-decls` would be lie,
but `-implicitly-trusted-extern-fn-decls` I can tolerate.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 01:16:49 UTC, Walter Bright wrote: On 5/24/2020 5:56 PM, Timon Gehr wrote: It's only greenwashing if it's misleading. Putting @safe is a lie, putting @trusted is honest. It is not honest unless the programmer actually carefully examined the interface and the documentation to determine if it is a safe interface or not. For example, labeling memcpy() with @trusted is not honest. Forcing people to add uncheckable annotations is a path to convenience, not honesty. What is the difference of @safe to @trusted in that respect? Does the compiler "carefully examines" any interface or documentation? Why not simply introducing new label as a solution, something in the realm @extern_safe_dont_know?
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: On 5/25/2020 7:04 PM, Johannes Loher wrote: Now let's compare the two different options: 1. With DIP1028 in its current form, the code will compile and a memory corruption will actually happen. The problem might be extremely difficult to track down for the developer because he has no clues whatsoever where to start looking. 2. With one of the solutions that were presented, the code will not compile as it is. According to your argument of „convenience“, the developer will probably just mark the function incorrectly as @trusted which makes the code compile. The memory corruption will happen. However, even if the developer did not think much about potential safety issues when adding @trusted to the function, he now still remembers that he did that (it was a conscious decision, even if it was a careless and lazy one). He has a clear point to start looking for the reason of the memory corruption. Do you honestly think option 1 is better? Yes, for reasons I carefully laid out. A direct response to Andre's thorough critique of your reasoning would be appreciated. > no clues whatsoever He can look at unattributed declarations. Again, as many have noted, putting a coverage problem like this on the programmer is problematic. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts. Greenwashing is bad. En masse greenwashing by the compiler, as mandated by the DIP currently, is really bad.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote: On 5/25/2020 7:04 PM, Johannes Loher wrote: [..] Do you honestly think option 1 is better? Yes, for reasons I carefully laid out. which fails to convince anyone because the reasoning is flawed. > no clues whatsoever He can look at unattributed declarations. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts. Unattended automatic greenwashing by the compiler is WORSE!
Re: DIP1028 - Rationale for accepting as is
On 5/25/2020 7:04 PM, Johannes Loher wrote: Now let's compare the two different options: 1. With DIP1028 in its current form, the code will compile and a memory corruption will actually happen. The problem might be extremely difficult to track down for the developer because he has no clues whatsoever where to start looking. 2. With one of the solutions that were presented, the code will not compile as it is. According to your argument of „convenience“, the developer will probably just mark the function incorrectly as @trusted which makes the code compile. The memory corruption will happen. However, even if the developer did not think much about potential safety issues when adding @trusted to the function, he now still remembers that he did that (it was a conscious decision, even if it was a careless and lazy one). He has a clear point to start looking for the reason of the memory corruption. Do you honestly think option 1 is better? Yes, for reasons I carefully laid out. > no clues whatsoever He can look at unattributed declarations. The whole debate boils down to "is greenwashing better, more honest, more debuggable than leaving things unattributed?" No on all three accounts.
Re: DIP1028 - Rationale for accepting as is
On Tuesday, 26 May 2020 at 01:16:49 UTC, Walter Bright wrote: On 5/24/2020 5:56 PM, Timon Gehr wrote: It's only greenwashing if it's misleading. Putting @safe is a lie, putting @trusted is honest. It is not honest unless the programmer actually carefully examined the interface and the documentation to determine if it is a safe interface or not. For example, labeling memcpy() with @trusted is not honest. Forcing people to add uncheckable annotations is a path to convenience, not honesty. This has already been repeated over and over again but I'll do it once more: If people cannot check the code (or get enough confidence in another way), they really shouldn’t annotate it with @trusted but with @system instead. If they annotate it with @trusted anyways, it is their own fault. And at least in that case they are lying and not the compiler. Also if something bad actually happens, they can at least search for „high risk areas“ (@trusted declarations). Just consider the following situation: A (maybe inexperienced) programmer needs to call an existing C function from D code. Because no bindings exist yet, he creates them himself. He does not add any annotations regarding safety. Now he uses this function in @safe code and he uses it in a way that actually creates a memory corruption (the function is not safe, but the developer does not know about that). Now let's compare the two different options: 1. With DIP1028 in its current form, the code will compile and a memory corruption will actually happen. The problem might be extremely difficult to track down for the developer because he has no clues whatsoever where to start looking. 2. With one of the solutions that were presented, the code will not compile as it is. According to your argument of „convenience“, the developer will probably just mark the function incorrectly as @trusted which makes the code compile. The memory corruption will happen. However, even if the developer did not think much about potential safety issues when adding @trusted to the function, he now still remembers that he did that (it was a conscious decision, even if it was a careless and lazy one). He has a clear point to start looking for the reason of the memory corruption. Do you honestly think option 1 is better?
Re: DIP1028 - Rationale for accepting as is
On 5/24/2020 5:56 PM, Timon Gehr wrote: It's only greenwashing if it's misleading. Putting @safe is a lie, putting @trusted is honest. It is not honest unless the programmer actually carefully examined the interface and the documentation to determine if it is a safe interface or not. For example, labeling memcpy() with @trusted is not honest. Forcing people to add uncheckable annotations is a path to convenience, not honesty.
Re: DIP1028 - Rationale for accepting as is
Wowza, wow. The voice of reason returns!
Re: DIP1028 - Rationale for accepting as is
The DIP is trying to accomplish (copied from its Rationale): A. Costs of unsafe code have become ever more apparent and expensive, and @safe has grown more capable. Users expect safety to be opt-out, not opt-in. B. Most code should be naturally safe, and system code should be relatively rare. It makes sense that the common case - safe code - should be the default and not require an annotation. Any judgment on the regime of extern(C) functions should be judged as follows: "To what extent does the chosen regime of C functions push the rationale forward?" The decision fails to support (A) because it keeps safety of C functions opt-in. It also fails to support (B) because it keeps code that does not require an annotation yet is not safe. A good process would simply go back to the drawing board once the matter is exposed. It is very easy for reasonable people to agree that there is no part of rationale that justifies the decision. Either redefine the rationale (adding e.g. "keep some applications compilable without change" as an important goal), or change the handing of C functions. The core argument recently detailed is: A. Define "greenwashing" as a quick workaround to get code to work. B. Describe in excruciating detail the experience with Java exceptions. C. Establish a vague analogy with people placing @trusted on C declarations. D. Assume without stating that the user would do "greenwashing" but the compiler doing it surreptitiously is somehow exempt. E. Conclude that extern(C) code must be trusted. This argument is already a smorgasbord of fallacies. It iterates a good subset of https://en.wikipedia.org/wiki/List_of_fallacies, and I am not kidding when I say a few edits to those pages with backreferences to this thread would be appropriate. To start: False analogy: making extern(C) function system is analogous with the Java exeception specification debacle. https://en.wikipedia.org/wiki/Argument_from_analogy#False_analogy This is a false analogy belying what seems to be a misunderstanding of the problem with Java exceptions: it was a maintenance problem, not an introduction problem. The matter was, during maintenance, routinely code would change the set of exceptions they threw. So maintainers would need to touch code that didn't belong to them. There was also no "shrinking" of exception specification lists because the compiler does not enforce larger-than-necessary lists. So these lists would just grow forever during maintenance, with no real information attached and no usefulness. None of these apply to @safe. Functions don't change their safety with ease, and when they do you definitely want to be informed. There's no growth of attributes for obvious reasons. To the extent the analogy applies, the burden of proof is entirely on the DIP, and in this case the differences are so many and large, it's not worth making the argument by analogy in the first place. The excruciatingly long discussion of Java exception specifications is an instance of another fallacy, misleading vividness: http://nizkor.com/features/fallacies/misleading-vividness.html It was expected that any of the bad things about exceptions would count as yet another good argument for the DIP. This constructs a textbook straw man: https://en.wikipedia.org/wiki/Straw_man So the argument constructs a straw man (Java exception specifications), discusses it at length on a long tangent (misleading vividness), to then conclude (by false analogy) that we don't want extern(C) functions to be @system. What's missing is, of course, an explanation on how the decision supports the very rationale of the DIP. Which it doesn't - it factually works straight against it. I decided to make this public only once the second boot fell (read on): On 5/23/20 11:28 PM, Walter Bright wrote: I'd like to emphasize: 1. It is not possible for the compiler to check any declarations where the implementation is not available. Not in D, not in any language. Declaring a declaration safe does not make it safe. Strawman: https://en.wikipedia.org/wiki/Straw_man Nobody asked for that, and listing it as an argument is pointing at a strawman. 2. If un-annotated declarations cause a compile time error, it is highly likely the programmer will resort to "greenwashing" - just slapping @safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've all done it. Sometimes even for good reasons. Nirvana fallacy: https://en.wikipedia.org/wiki/Nirvana_fallacy "Electric cars still use polluting materials and processes, so let's continue making gasoline cars." "People will add workarounds to code, so let's accept the dangerous code anyway." 3. Un-annotated declarations are easily detectable in a code review. Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion (Um... there's no grep for "find me all un-annnotated declarations") 4. Greenwashing is not easily detectable
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 19:44:00 UTC, Paul Backus wrote: On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote: I really appreciate your contributions. (The latter of which will be quite handy.) But isn't the whole point of Walter's implementation of @safe-as-default is that FFIs are going to be assumed @safe? So they should still compile; that is unless you're referring to the need for re-evaluation. Yes, that's Walter's current plan. I am hoping that he'll change his mind about allowing un-annotated FFIs to be assumed @safe, but even if he doesn't, having the compiler print a warning will allow those of us who care about safety to find them and handle them properly. I'm still unsure which implementation is the right one, but I think this middle-ground you're attempting to get merged is a good compromise. So, again, thank you!
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote: I share your concerns on this, but disagree on the likelihood of reviews having gone by under the assumption of @system by default. I doubt most people even thought about @safe/@trusted/@system, and that's assuming anyone reviewed the code in the first place. A few years ago I submitted several PRs to Phobos to mark all unittests that could with @safe explicitly. I'd say that was a good example of nobody reviewing them for their @systemness. I don't think a community run project without a lead or team behind it is a very good example. No one is going to spend their free time reviewing code ensuring it is memory safe. I've seen it such that unit tests in phobos/druntime weren't being run on some platforms, and there were tests that failed when run.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote: I really appreciate your contributions. (The latter of which will be quite handy.) But isn't the whole point of Walter's implementation of @safe-as-default is that FFIs are going to be assumed @safe? So they should still compile; that is unless you're referring to the need for re-evaluation. Yes, that's Walter's current plan. I am hoping that he'll change his mind about allowing un-annotated FFIs to be assumed @safe, but even if he doesn't, having the compiler print a warning will allow those of us who care about safety to find them and handle them properly.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote: Even if you did review your code *and* all of your dependencies, if you did it before DIP 1028 was accepted (i.e., any time in the last 10 years or so), you may have deliberately left external @system functions un-annotated, because @system was the default. In fact, I *know* that people have done this, because I've been running my patched version of the compiler [1] against the dmd, druntime, and phobos codebases, and I have found an enormous number of such un-annotated declarations. Here are just a few of them in druntime: https://github.com/dlang/druntime/pull/3117 Lots and lots of projects in the D ecosystem are going to need similar patches to be compatible with DIP 1028. And I don't think it's realistic to expect people to make these changes without some kind of help from the tooling. [1] https://github.com/dlang/dmd/pull/11176 I really appreciate your contributions. (The latter of which will be quite handy.) But isn't the whole point of Walter's implementation of @safe-as-default is that FFIs are going to be assumed @safe? So they should still compile; that is unless you're referring to the need for re-evaluation.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 18:39:16 UTC, mw wrote: Can we declare the new D compiler version 3.0 for @safe-as-default? and keep version 2.x compiler for the old @system-as-default? @safe-by-default is currently hidden behind a -preview switch. I think it's safe to assume there will be a long deprecation period before it's made the default, and even then, there will be a -revert switch to turn it off.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote: Even if you did review your code *and* all of your dependencies, if you did it before DIP 1028 was accepted (i.e., any time in the last 10 years or so), you may have deliberately left external @system functions un-annotated, because @system was the default. In fact, I *know* that people have done this, because I've been running my patched version of the compiler [1] against the dmd, druntime, and phobos codebases, and I have found an enormous number of such un-annotated declarations. Here are just a few of them in druntime: https://github.com/dlang/druntime/pull/3117 Lots and lots of projects in the D ecosystem are going to need similar patches to be compatible with DIP 1028. And I don't think it's realistic to expect people to make these changes without some kind of help from the tooling. [1] https://github.com/dlang/dmd/pull/11176 Can we declare the new D compiler version 3.0 for @safe-as-default? and keep version 2.x compiler for the old @system-as-default? In industries, it is a big-NO-NO to break exiting *working* code base. I've read some past threads talking about the instability of compilers (due to new feature introduced), e.g. in this on_leaving_d thread: https://forum.dlang.org/thread/ncbawciyybdksecur...@forum.dlang.org?page=1 https://gitlab.com/mihails.strasuns/blog/blob/master/articles/on_leaving_d.md Let me quote what D's *industry users* (whose trademark is on the front page of dlang.org) have said 2 years ago: """ But when it comes to be boring and predictable, D fails terribly. You can't assume that released features will become 100% usable (no bugs, sufficient docs and tests, well-defined integration with other language features) even few years after initial release. You can't assume that next compiler upgrade won't suddenly break your project or any of its transitive dependencies. You can't assume that any compiler version will be maintained for more than few months. You can't assume there is any control over how declared vision documents get executed in practice. You can't trust any promises from language authors because they don't keep any track of those. It is anarchy driven development in all its glory. """ I understand the desire of Walter to continue innovation and I appreciate it, and I also think this change from @system to @safe-as default is an important step for D's future direction, as I commented in my thread yesterday: https://forum.dlang.org/post/brvexgjuqkbebizdm...@forum.dlang.org """ -- the recent change @safe as default opt-in is good thing: without a @nogc phobos, nobody is going to write @system level software using D. I think this @safe change means the shift of D's focus from system software to application software. And I'm glad to see the the template project that dub generate starts with "app.d", which is the right direction for D. """ But on the other hand, we also need to take care of our existing D users, esp industry users, are they going to be happy about breaking their working code base with a new compiler version? For example, we can define a metric: say, how many percent of the packages on https://code.dlang.org/ will continue work as-it-is *without* to make any change with the new @safe-as-default compiler. If > 95% of these packages (the 68–95–99.7 three-sigma rule of thumb https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule) require not any code change, then go ahead, release the new @safe-as-default compiler as 2.093. Otherwise, I'd suggest to keep 2 versions of compiler: -- version 2.x as stable compiler for (industry) users to use in production, and will only provide bug fixes, but no new language feature (changes) to the compiler. (And I believe the 2.x version D language has already be *feature rich enough* to beat all those Java, C#, and Rust for people to use in production software). -- version 3.x for Walter to experiment and explore new ideas, and warn all the users that version 3 is unstable compiler feature-wise, use-it-at-your-own-risk, your code base may need major change in the future. Thoughts?
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 17:09:50 UTC, Clarice wrote: On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote: the compiler must warn D programmers that their declarations need to be reviewed for compatibility with @safe-by-default. Whether they were reviewed before that or not makes no difference. I'm not an engineer; I'm not learned in CS. So some clarification on this would be nice: if you're pulling in C code, or any code written in an unsafe language, wouldn't the developers doing such a thing already be aware they need to do their due diligence? Foreign code isn't just going to magically appear in your codebase, right? Maybe through dependencies, but one should check those too, eh? (And if the dependency tree is needlessly large, then the language has another, maybe bigger, problem à la the NPM ecosystem.) Even if you did review your code *and* all of your dependencies, if you did it before DIP 1028 was accepted (i.e., any time in the last 10 years or so), you may have deliberately left external @system functions un-annotated, because @system was the default. In fact, I *know* that people have done this, because I've been running my patched version of the compiler [1] against the dmd, druntime, and phobos codebases, and I have found an enormous number of such un-annotated declarations. Here are just a few of them in druntime: https://github.com/dlang/druntime/pull/3117 Lots and lots of projects in the D ecosystem are going to need similar patches to be compatible with DIP 1028. And I don't think it's realistic to expect people to make these changes without some kind of help from the tooling. [1] https://github.com/dlang/dmd/pull/11176
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote: the compiler must warn D programmers that their declarations need to be reviewed for compatibility with @safe-by-default. Whether they were reviewed before that or not makes no difference. I'm not an engineer; I'm not learned in CS. So some clarification on this would be nice: if you're pulling in C code, or any code written in an unsafe language, wouldn't the developers doing such a thing already be aware they need to do their due diligence? Foreign code isn't just going to magically appear in your codebase, right? Maybe through dependencies, but one should check those too, eh? (And if the dependency tree is needlessly large, then the language has another, maybe bigger, problem à la the NPM ecosystem.)
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote: A few years ago I submitted several PRs to Phobos to mark all unittests that could with @safe explicitly. I'd say that was a good example of nobody reviewing them for their @systemness. Ideally you should be able to blindly mark every function definition with @safe, because the compiler will catch you if you fall. Only if you type @trusted you should need to be careful.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote: On Sunday, 24 May 2020 at 16:44:01 UTC, Paul Backus wrote: If we were designing a new language from scratch, I would agree 100% with your reasoning. The problem is that there are un-annotated declarations in existing code that have already been reviewed, committed, and published under the assumption of @system-by-default. Those declarations need to be flagged for re-review in order to avoid introducing silent safety violations to existing D projects. I share your concerns on this, but disagree on the likelihood of reviews having gone by under the assumption of @system by default. I doubt most people even thought about @safe/@trusted/@system, and that's assuming anyone reviewed the code in the first place. A few years ago I submitted several PRs to Phobos to mark all unittests that could with @safe explicitly. I'd say that was a good example of nobody reviewing them for their @systemness. Walter's claim was that "un-annotated declarations are easily detectable in code review." The intent of my response was simply to point out that for existing D code, the opportunity for such review, and therefore for the detection of such declarations, is already in the past. In any case, your response does not change my conclusion: the compiler must warn D programmers that their declarations need to be reviewed for compatibility with @safe-by-default. Whether they were reviewed before that or not makes no difference.
Re: DIP1028 - Rationale for accepting as is
On Sunday, 24 May 2020 at 16:44:01 UTC, Paul Backus wrote: On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote: I'd like to emphasize: 1. It is not possible for the compiler to check any declarations where the implementation is not available. Not in D, not in any language. Declaring a declaration safe does not make it safe. 2. If un-annotated declarations cause a compile time error, it is highly likely the programmer will resort to "greenwashing" - just slapping @safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've all done it. Sometimes even for good reasons. 3. Un-annotated declarations are easily detectable in a code review. [...] If we were designing a new language from scratch, I would agree 100% with your reasoning. The problem is that there are un-annotated declarations in existing code that have already been reviewed, committed, and published under the assumption of @system-by-default. Those declarations need to be flagged for re-review in order to avoid introducing silent safety violations to existing D projects. I share your concerns on this, but disagree on the likelihood of reviews having gone by under the assumption of @system by default. I doubt most people even thought about @safe/@trusted/@system, and that's assuming anyone reviewed the code in the first place. A few years ago I submitted several PRs to Phobos to mark all unittests that could with @safe explicitly. I'd say that was a good example of nobody reviewing them for their @systemness.
Re: DIP1028 - Rationale for accepting as is
On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote: The more I think of Atila's and Walter's responses, the more they are starting to make sense. [...] Thank you for the anecdote, especially since it captures the spirit of what I've been trying to convey here.
Re: DIP1028 - Rationale for accepting as is
On Friday, 22 May 2020 at 20:08:37 UTC, Dukc wrote:
On Friday, 22 May 2020 at 18:27:42 UTC, Atila Neves wrote:
Sorry, I didn't express myself well. I meant that the user can
still mark functions as @system, they just have to do it
explicitly.
Hm, DPP might be of help here. Becuse I quess you are going to
make sure it'll mark everything `@system`?
My plan is probably to do that, yes. I've been thinking about it
and I'll likely need to give users options to opt-in to declaring
functions @safe, otherwise none of them every would be with this:
@system {
#include "header.h"
}
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 15:07:19 UTC, Petar Kirov [ZombineDev] wrote: I don't want to change the definition of @safe in D, but would rather like if D supported @strongSafe, that interested people like me could opt into. I know that worded like this it may sound like too narrow feature to add to the language (or at least not having favorable complexity/use cases ratio). So instead, I'd like to have transitive UDAs [1], a feature that has been requested by many, for various use cases ;) [1]: Basically I want to be able to implement function attributes like @nogc or nothrow in user-space, but that's a long way from now, as first, we need to be able to introspect function bodies. Sounds to me like a great use-case for the DMD-frontend-as-a-library project.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 13:43:07 UTC, Paul Backus wrote: On Monday, 25 May 2020 at 13:22:36 UTC, Petar Kirov [ZombineDev] wrote: On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov [ZombineDev] wrote: It may be true (of course modulo meta-programming) that it doesn't make a difference for the calling code, but I personally want have the guarantees that a function that I'm doesn't make a difference for the calling code, but personally I want [to] have the guarantees that a function that I'm calling is truly @safe (it doesn't contain or call any @trusted code, transitively, nor it calls any @safe code, which access global variables initialized by @system static/module constructors). This is very far from a rigorous definition of "strong @safe-ty" - but I hope it's just enough for the casual reader to understand my intention. I'm sure this is reasonable for your use-case, but I hope you can recognize that this definition of safety is far too narrow to be suitable for a general-purpose programming language (which D purports to be). Most people would like their @safe code to be able to do I/O, for example, despite the fact that it necessarily involves calling @system code under the hood. I don't want to change the definition of @safe in D, but would rather like if D supported @strongSafe, that interested people like me could opt into. I know that worded like this it may sound like too narrow feature to add to the language (or at least not having favorable complexity/use cases ratio). So instead, I'd like to have transitive UDAs [1], a feature that has been requested by many, for various use cases ;) [1]: Basically I want to be able to implement function attributes like @nogc or nothrow in user-space, but that's a long way from now, as first, we need to be able to introspect function bodies.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 13:22:36 UTC, Petar Kirov [ZombineDev] wrote: On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov [ZombineDev] wrote: It may be true (of course modulo meta-programming) that it doesn't make a difference for the calling code, but I personally want have the guarantees that a function that I'm doesn't make a difference for the calling code, but personally I want [to] have the guarantees that a function that I'm calling is truly @safe (it doesn't contain or call any @trusted code, transitively, nor it calls any @safe code, which access global variables initialized by @system static/module constructors). This is very far from a rigorous definition of "strong @safe-ty" - but I hope it's just enough for the casual reader to understand my intention. I'm sure this is reasonable for your use-case, but I hope you can recognize that this definition of safety is far too narrow to be suitable for a general-purpose programming language (which D purports to be). Most people would like their @safe code to be able to do I/O, for example, despite the fact that it necessarily involves calling @system code under the hood.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov [ZombineDev] wrote: It may be true (of course modulo meta-programming) that it doesn't make a difference for the calling code, but I personally want have the guarantees that a function that I'm doesn't make a difference for the calling code, but personally I want [to] have the guarantees that a function that I'm calling is truly @safe (it doesn't contain or call any @trusted code, transitively, nor it calls any @safe code, which access global variables initialized by @system static/module constructors). This is very far from a rigorous definition of "strong @safe-ty" - but I hope it's just enough for the casual reader to understand my intention. In my line work (blockchain smart contracts) some of the ways of how this is typically achieved include: * having a very minimal smart contract code size * having either no third-party dependencies or using one or two which are open-source and more importantly verified by multiple teams and having very high reputation * extensive code auditing by third-party teams. Depending on the circumstances, we may end up paying more for the auditing of the code, than the actual development. That said, there is no "strong"-@safe today and even if there That said, there is no "strong-@safe" [in D] today and even if there was, it would account for a tiny subset of all attack vectors that I have to care about (basically all possible logical bugs allowed in type-safe and memory-safe code), but I'm not sure how erasing the difference between @safe and @trusted on the interface level would help.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 12:41:01 UTC, Paul Backus wrote: On Monday, 25 May 2020 at 12:30:11 UTC, Zoadian wrote: On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote: It is meant to mean that at some point it has been mechanically checked by the compiler. Either during current compilation or a prior one. Which means it has to be valid on function declarations without bodies so i.e. .di file generation works correctly which is just a generated D file, nothing special syntax of semantics wise. .di files _could_ just use @trusted instead of @safe. but for extern(D) we could at least add it to the name mangling. it's still not 100% safe, but at least you'd have to work hard to get it wrong. It's been proposed before that @safe and @trusted should have the same mangling, since there's no difference between them from the calling code's perspective. It may be true (of course modulo meta-programming) that it doesn't make a difference for the calling code, but I personally want have the guarantees that a function that I'm calling is truly @safe (it doesn't contain or call any @trusted code, transitively, nor it calls any @safe code, which access global variables initialized by @system static/module constructors). In my line work (blockchain smart contracts) some of the ways of how this is typically achieved include: * having a very minimal smart contract code size * having either no third-party dependencies or using one or two which are open-source and more importantly verified by multiple teams and having very high reputation * extensive code auditing by third-party teams. Depending on the circumstances, we may end up paying more for the auditing of the code, than the actual development. That said, there is no "strong"-@safe today and even if there was, it would account for a tiny subset of all attack vectors that I have to care about (basically all possible logical bugs allowed in type-safe and memory-safe code), but I'm not sure how erasing the difference between @safe and @trusted on the interface level would help.
Re: DIP1028 - Rationale for accepting as is
On 25.05.20 14:22, Johannes T wrote: Sorry, I phrased it poorly. I meant @trusted would be used more frequently. The focus would spread and lead to less rigorous checks. The focus is as wide as it's ever been. If you want to verify a program, you have to check those prototypes, whether they're explicitly @trusted or implicitly @safe. @safe-by-default only makes them harder to find.
Re: DIP1028 - Rationale for accepting as is
On 25.05.20 14:22, Johannes T wrote: On Monday, 25 May 2020 at 11:52:27 UTC, Timon Gehr wrote: On 25.05.20 11:25, Johannes T wrote: @trusted can't be trusted That's the point of @trusted. ._. The code is trusted by the programmer, not the annotation by the compiler. Sorry, I phrased it poorly. I meant @trusted would be used more frequently. The focus would spread and lead to less rigorous checks. This is just not true. If the compiler forces you to decide to put either @trusted or @system, then you will not get more usage of @trusted than if it implicitly decides for you that what you want is implicit @trusted.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote: On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote: [..] But with the DIP in its current form, we make @safe lose its meaning and power, which is much worse in my opinion. [..] The alternative, not making extern @safe, would result in more untrustworthy @trusted code we have to worry about. It's a vicious circle. Wrong. The quantity of untrustworthy code remains the same, but with DIP1028 (at least in the current form) the compiler sweeps the previously @system code under the rug and makes it harder for those who care about safety to trust @safe. @safe must mean only one thing: compiler verified. Or otherwise needing less manual review. @system and @trusted means that code review should be prioritized. @safe non-extern (D) marked either by the programmer, or the implicitly by the compiler should be disallowed as it *is* greenwashing. I try to relax my view on extern annotations. They are @system. We *should* go ahead and diligently mark with @trusted. From experience, it doesn't normally happen. It didn't happen, because it didn't need to. Naturally, most things go through the path of least resistance. Most developers are coming from other languages where they have never had the requirement to write @safe code [external pressure]. Also previously, @safe wasn't the default for D function definitions, so there was less [internal pressure] to do so. With @safe being the default of function definitions, it's more difficult to leave code as @system (of course, modulo @trusted). I don't like @safe extern, but it seems like the lesser evil. No, @safe extern is the worst possible option! It basically makes @safe meaningless. Walter got a lot of flak. I tried to retrace his thoughts and see the merits. On several occasions (e.g. on Reddit) I have defended Walter from unfair accusations, however, in this case, he's rightfully criticized. He seems to think that he's taking an unpopular decision for the greater good, but that's not the case. @safe-by-default on D function definitions could be considered an unpopular decision for the greater good. Implicitly @safe non-extern(D) functions is greenwashing, where the responsibility for the action is removed from the developer by a compiler switch. That's basically negating all the benefits of @safe-by-default on function definitions.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 12:30:11 UTC, Zoadian wrote: On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote: It is meant to mean that at some point it has been mechanically checked by the compiler. Either during current compilation or a prior one. Which means it has to be valid on function declarations without bodies so i.e. .di file generation works correctly which is just a generated D file, nothing special syntax of semantics wise. .di files _could_ just use @trusted instead of @safe. but for extern(D) we could at least add it to the name mangling. it's still not 100% safe, but at least you'd have to work hard to get it wrong. It's been proposed before that @safe and @trusted should have the same mangling, since there's no difference between them from the calling code's perspective.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 12:22:25 UTC, Zoadian wrote: [..] there is no such thing as a trustworthy @trusted. [..] Sorry, my bad. I meant the quality of @trusted annotations would decline.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote: On 25/05/2020 10:29 PM, Zoadian wrote: you complain about @trusted losing it's meaning, but @safe was ment to mean "mechanically verified memory safety". it should be forbidden to add @safe to any function that can not be verified by the compiler. It is meant to mean that at some point it has been mechanically checked by the compiler. Either during current compilation or a prior one. Which means it has to be valid on function declarations without bodies so i.e. .di file generation works correctly which is just a generated D file, nothing special syntax of semantics wise. .di files _could_ just use @trusted instead of @safe. but for extern(D) we could at least add it to the name mangling. it's still not 100% safe, but at least you'd have to work hard to get it wrong.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote: On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote: [..] But with the DIP in its current form, we make @safe lose its meaning and power, which is much worse in my opinion. [..] The alternative, not making extern @safe, would result in more untrustworthy @trusted code we have to worry about. It's a vicious circle. I try to relax my view on extern annotations. They are @system. We *should* go ahead and diligently mark with @trusted. From experience, it doesn't normally happen. I don't like @safe extern, but it seems like the lesser evil. Walter got a lot of flak. I tried to retrace his thoughts and see the merits. From my perspective it is really simple: Either we have a strict @safety system (with exactly one escape hatch) or we don't. At the moment we don't because you can have @safe extern(C) declarations but with this DIP it becomes even worse as it is more likely that @safe promises are broken without anybody noticing. If you and more importantly Walter and Atila really think this is the lesser evil and that it is worth dropping the strict @safety system with exactly one controlled escape hatch for this, then we also need to clarify what @safe means after that. We can definitely not claim that it means machine verified except for @trusted because extern(C) functions are another escape hatch. The question has been asked in this thread a few times already: What does @safe actually mean? How to „sell“ / „advertise“ it?
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 12:22:25 UTC, Zoadian wrote: there is no such thing as a trustworthy @trusted. not with how the whole safety system work now. you can break previously verified @trusted code by just writing @safe code today. Do you have an example of code that was correctly labeled @trusted in DMD version X, which may not be labeled @trusted in DMD version Y because of changes to @safe/@trusted@/@system?
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote: On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote: [..] But with the DIP in its current form, we make @safe lose its meaning and power, which is much worse in my opinion. [..] The alternative, not making extern @safe, would result in more untrustworthy @trusted code we have to worry about. It's a vicious circle. I try to relax my view on extern annotations. They are @system. We *should* go ahead and diligently mark with @trusted. From experience, it doesn't normally happen. I don't like @safe extern, but it seems like the lesser evil. Walter got a lot of flak. I tried to retrace his thoughts and see the merits. there is no such thing as a trustworthy @trusted. not with how the whole safety system work now. you can break previously verified @trusted code by just writing @safe code today.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 11:52:27 UTC, Timon Gehr wrote: On 25.05.20 11:25, Johannes T wrote: @trusted can't be trusted That's the point of @trusted. ._. The code is trusted by the programmer, not the annotation by the compiler. Sorry, I phrased it poorly. I meant @trusted would be used more frequently. The focus would spread and lead to less rigorous checks.
Re: DIP1028 - Rationale for accepting as is
On 25.05.20 11:25, Johannes T wrote: @trusted can't be trusted That's the point of @trusted. ._. The code is trusted by the programmer, not the annotation by the compiler.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote: [..] But with the DIP in its current form, we make @safe lose its meaning and power, which is much worse in my opinion. [..] The alternative, not making extern @safe, would result in more untrustworthy @trusted code we have to worry about. It's a vicious circle. I try to relax my view on extern annotations. They are @system. We *should* go ahead and diligently mark with @trusted. From experience, it doesn't normally happen. I don't like @safe extern, but it seems like the lesser evil. Walter got a lot of flak. I tried to retrace his thoughts and see the merits.
Re: DIP1028 - Rationale for accepting as is
On 25/05/2020 10:29 PM, Zoadian wrote: you complain about @trusted losing it's meaning, but @safe was ment to mean "mechanically verified memory safety". it should be forbidden to add @safe to any function that can not be verified by the compiler. It is meant to mean that at some point it has been mechanically checked by the compiler. Either during current compilation or a prior one. Which means it has to be valid on function declarations without bodies so i.e. .di file generation works correctly which is just a generated D file, nothing special syntax of semantics wise.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 09:25:52 UTC, Johannes T wrote: On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote: [..] After thinking about it, Walter ultimately made the right decision, leading to overall higher safety and code quality. We all agree that making extern C @safe is incorrect. It's also meaningless. Even if you were to verify the safety of a specific version of your binding, it can't be known what's loaded at runtime. It's not the compiler's concern. @safe extern shall be an error. We might additionally make an exception and make all extern C @system. It would be correct for the declarations, but inconsistent in regard to default safety. It doesn't affect the outcome. Let's say we'll go with @system, it gives us a bit more freedom. We hit compile, our now safe wrappers are errors. We are most likely to do one of two things, depending on the module failing to compile. If it mostly wrappers, we slap @trusted: at the top. If there are just a few functions calling C, we only annotate those with @trusted. Let's be real, we probably won't begin checking and annotating the trustworthiness of the C functions. An individual programmer might, but not on average. We are formally correct, but @trusted can't be trusted at this point. It has lost its meaning. We now have to check all foreign @trusted code, which we probably won't. We could have moved the problem one level down and slapped @trusted: on top of the @system declarations. Now there is a bunch of safe code using them. It doesn't change much. @trusted: is easier to grep, but we won't put everything down and begin trustworthily annotate extern C. So yeah, I do believe Walter was right. @safe on extern is formally incorrect but leads to safer code because @trusted still has its power. you complain about @trusted losing it's meaning, but @safe was ment to mean "mechanically verified memory safety". it should be forbidden to add @safe to any function that can not be verified by the compiler. the compiler should do it's best to alert you of any mistakes you might make. if the compiler does silently add @safe or @trusted to extern(C) functions you are are to miss annotating it. that is a bad thing for everyone that actually cares about safety. you can't prevent programmers from doing stupid things. but you can at least warn them.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 09:25:52 UTC, Johannes T wrote: On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote: [..] After thinking about it, Walter ultimately made the right decision, leading to overall higher safety and code quality. We all agree that making extern C @safe is incorrect. It's also meaningless. Even if you were to verify the safety of a specific version of your binding, it can't be known what's loaded at runtime. It's not the compiler's concern. @safe extern shall be an error. We might additionally make an exception and make all extern C @system. It would be correct for the declarations, but inconsistent in regard to default safety. It doesn't affect the outcome. Let's say we'll go with @system, it gives us a bit more freedom. We hit compile, our now safe wrappers are errors. We are most likely to do one of two things, depending on the module failing to compile. If it mostly wrappers, we slap @trusted: at the top. If there are just a few functions calling C, we only annotate those with @trusted. Let's be real, we probably won't begin checking and annotating the trustworthiness of the C functions. An individual programmer might, but not on average. We are formally correct, but @trusted can't be trusted at this point. It has lost its meaning. We now have to check all foreign @trusted code, which we probably won't. We could have moved the problem one level down and slapped @trusted: on top of the @system declarations. Now there is a bunch of safe code using them. It doesn't change much. @trusted: is easier to grep, but we won't put everything down and begin trustworthily annotate extern C. So yeah, I do believe Walter was right. @safe on extern is formally incorrect but leads to safer code because @trusted still has its power. So basically you are saying we should do it the way it is described in the DIP because otherwise it will lead developers to incorrect usage of @trusted (just slapping it on declarations whiteout actually checking the implementation) thus making it loose its „power“ which leads to less safe code. But with the DIP in its current form, we make @safe lose its meaning and power, which is much worse in my opinion. It makes the statement „@safe code cannot create memory corruptions except for @trusted code“ wrong (it already is wrong now but it really shouldn’t be!). What you are describing is really just an admit of defeat. Then why still bother with @safe at all if it doesn't give us any meaningful guarantees? I don't think we are at that point (yet).
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote: [..] After thinking about it, Walter ultimately made the right decision, leading to overall higher safety and code quality. We all agree that making extern C @safe is incorrect. It's also meaningless. Even if you were to verify the safety of a specific version of your binding, it can't be known what's loaded at runtime. It's not the compiler's concern. @safe extern shall be an error. We might additionally make an exception and make all extern C @system. It would be correct for the declarations, but inconsistent in regard to default safety. It doesn't affect the outcome. Let's say we'll go with @system, it gives us a bit more freedom. We hit compile, our now safe wrappers are errors. We are most likely to do one of two things, depending on the module failing to compile. If it mostly wrappers, we slap @trusted: at the top. If there are just a few functions calling C, we only annotate those with @trusted. Let's be real, we probably won't begin checking and annotating the trustworthiness of the C functions. An individual programmer might, but not on average. We are formally correct, but @trusted can't be trusted at this point. It has lost its meaning. We now have to check all foreign @trusted code, which we probably won't. We could have moved the problem one level down and slapped @trusted: on top of the @system declarations. Now there is a bunch of safe code using them. It doesn't change much. @trusted: is easier to grep, but we won't put everything down and begin trustworthily annotate extern C. So yeah, I do believe Walter was right. @safe on extern is formally incorrect but leads to safer code because @trusted still has its power.
Re: DIP1028 - Rationale for accepting as is
On 5/23/2020 5:13 AM, Steven Schveighoffer wrote: And let's be honest here, if you are OK with it, putting @trusted: at the top of your extern(C) functions is fine with me. At least that's not a lie. @trusted says the interface to the function is safe. If the programmer did not actually check the interface, it is greenwashing. For example, memcpy() does not have a safe interface and marking it @trusted doesn't improve matters.
Re: DIP1028 - Rationale for accepting as is
On Monday, 25 May 2020 at 01:04:24 UTC, Timon Gehr wrote: On 24.05.20 11:10, Walter Bright wrote: On 5/23/2020 11:26 PM, Bruce Carneal wrote: I don't believe that you or any other competent programmer greenwashes safety critical code. Regardless, the safety conscious must review their dependencies whatever default applies. That's the theory. But we do, for various reasons. I've seen it a lot over the years, at all levels of programming ability. It particularly happens when someone needs to get the code compiling and running, and the error message is perceived as a nuisance getting in the way. We should be very careful about adding nuisances to the language that make it easier to greenwash than to do the job correctly. Implicit greenwashing by the compiler is a nuisance that makes it harder to do the job correctly and easier to do the wrong thing. Yes, it would be a big nuisance. Absent a change in the DIP the safety conscious who want to continue with D will try to back out the compiler lies as best they can: additional tooling, additional code review strictures, selective rewrites, ... Not sure how that unfortunate future would play out exactly but it would not be pretty. Much much better to fix the DIP.
Re: DIP1028 - Rationale for accepting as is
On 24.05.20 11:10, Walter Bright wrote: On 5/23/2020 11:26 PM, Bruce Carneal wrote: I don't believe that you or any other competent programmer greenwashes safety critical code. Regardless, the safety conscious must review their dependencies whatever default applies. That's the theory. But we do, for various reasons. I've seen it a lot over the years, at all levels of programming ability. It particularly happens when someone needs to get the code compiling and running, and the error message is perceived as a nuisance getting in the way. We should be very careful about adding nuisances to the language that make it easier to greenwash than to do the job correctly. Implicit greenwashing by the compiler is a nuisance that makes it harder to do the job correctly and easier to do the wrong thing.
