Re: @trust is an encapsulation method, not an escape
On 2/9/15 10:13 AM, Marc =?UTF-8?B?U2Now7x0eiI=?= schue...@gmx.net wrote: On Monday, 9 February 2015 at 14:40:36 UTC, Steven Schveighoffer wrote: On 2/7/15 7:11 AM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= ola.fosheim.grostad+dl...@gmail.com wrote: You are trying to do this: 1. mechanically verify the whole @trusted region 2. manually verify the whole @trusted region, but be sloppy about it here an there 3. Ooops, we were sloppy in the wrong spot... No. A @trusted function is manually verified, period. But we also must tag potential points of leakage with @system. In fact, it probably could be a warning/error if you have a @trusted function without any @system escapes (it could just be marked @safe). That's a nice migration path, btw. First, warn about @trusted functions without @system blocks and don't enforce @safe-ty inside them, later disallow them and do enforce @safe-ty in the others. Yes, that solves the problem of breaking code with this... Nice idea. -Steve
Re: @trust is an encapsulation method, not an escape
On Monday, 9 February 2015 at 14:40:36 UTC, Steven Schveighoffer wrote: On 2/7/15 7:11 AM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= ola.fosheim.grostad+dl...@gmail.com wrote: You are trying to do this: 1. mechanically verify the whole @trusted region 2. manually verify the whole @trusted region, but be sloppy about it here an there 3. Ooops, we were sloppy in the wrong spot... No. A @trusted function is manually verified, period. But we also must tag potential points of leakage with @system. In fact, it probably could be a warning/error if you have a @trusted function without any @system escapes (it could just be marked @safe). That's a nice migration path, btw. First, warn about @trusted functions without @system blocks and don't enforce @safe-ty inside them, later disallow them and do enforce @safe-ty in the others.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 7:29 PM, H. S. Teoh via Digitalmars-d wrote: On Fri, Feb 06, 2015 at 04:04:48PM -0800, Walter Bright via Digitalmars-d wrote: [...] I agree with Andrei in that I do not believe that reviewing a @trusted function, line by line, for safety is necessarily some sort of maintenance nightmare. If it is, then a refactoring should be considered to encapsulate the unsafe code in a smaller, simpler manner. [...] This does not take into the account the fact that a @trusted function may call other, non-@trusted, functions. When one of those other functions changes, the @trusted function necessarily needs to be reviewed again. This problem isn't solved by the proposal, however. If you are calling a @system function inside a @trusted function, and you've marked it as @system, then changing the @system function does not affect the call. However, if you changed a @safe function into a @system function, then a call inside a @trusted function would have to now be annotated. It's important to note that our proposal will not fix cases where something subtle happens inside a @system block. What it DOES do is limit this effect to the @system block instead of the whole function. -Steve
Re: @trust is an encapsulation method, not an escape
On 2/6/15 4:48 PM, Walter Bright wrote: On 2/6/2015 11:11 AM, H. S. Teoh via Digitalmars-d wrote: This is precisely why I have lost all interest in @safe. It's clear that the present problematic situation will continue to hold, and the decision makers are not interested to address it. I am not going to waste any more time and energy on this topic. In this thread at 8:20PM last night, Dicebot asked me: I am not even sure how you can show the example though, to be honest - implied issues are about maintaining code and not just writing it. And I replied with a specific example of how to fix one aspect of std.array. There have been no replies. What else can I do to address it? In the case you bring up, maintenance is easy -- the code is incorrect, it needs to be fixed/rewritten. No solution ever implemented or proposed can stop this from happening. The case being discussed by Dicebot, and most of us, involves a case where an entire @trusted function is PROPERLY implemented, yet someone adds incorrect code to it. The compiler does not complain. Note that if the requested solution was implemented, each of these calls should be individual cases to inspect. I don't think anyone disagrees that uninitializedArray shouldn't be a public @trusted function. But individual cases of it CAN be properly safe. -Steve
Re: @trust is an encapsulation method, not an escape
On Monday, 9 February 2015 at 14:40:36 UTC, Steven Schveighoffer wrote: But we also must tag potential points of leakage with @system. In fact, it probably could be a warning/error if you have a @trusted function without any @system escapes (it could just be marked @safe). Think of it this way: the @system tags are the only places where issues can creep into the function. But then you have to apply the leaks to the whole function. It makes the problem of finding potential safety issues more tractable, because the compiler forces us to identify the root causes. The compiler should only verify what is needed for the type system to work. There is no need to differentiate between @trusted and @system for that. If you require using @system for annotation, then you will have to change the language every time you improve the verifier. Because this ad hoc annotation will be unsuitable for a more powerful verifier. And it isn't obvious that treating @trusted as @safe will not lead to false positives.
Re: @trust is an encapsulation method, not an escape
On 2/7/15 7:11 AM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= ola.fosheim.grostad+dl...@gmail.com wrote: You are trying to do this: 1. mechanically verify the whole @trusted region 2. manually verify the whole @trusted region, but be sloppy about it here an there 3. Ooops, we were sloppy in the wrong spot... No. A @trusted function is manually verified, period. But we also must tag potential points of leakage with @system. In fact, it probably could be a warning/error if you have a @trusted function without any @system escapes (it could just be marked @safe). Think of it this way: the @system tags are the only places where issues can creep into the function. But then you have to apply the leaks to the whole function. It makes the problem of finding potential safety issues more tractable, because the compiler forces us to identify the root causes. -Steve
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 00:31:41 UTC, H. S. Teoh wrote: This does not take into the account the fact that a @trusted function may call other, non-@trusted, functions. When one of those other functions changes, the @trusted function necessarily needs to be reviewed again. However, under the current implementation, this is not done because when the other, non-@trusted, function is modified, nobody thinks to re-review the @trusted function. They may not even be in the same module. There is no mechanism in place to raise a warning flag when a @trusted function's dependencies are modified. Thus, the proof of safety of the @trusted function has been invalidated, but trust continues to be conferred upon it. So what you should ask for is a way to sign @trusted with a timestamp that indicates when it was last proven to be safe, e.g.: @trusted(2015-01-01T12:30:12z) Then ask for a fine grained dependency tracking tool that can extract changes from git. Such a dependency tracking tool could be a nice stepping stone for faster compilation (sub-file-level recompilation). So there is synergies in this. The proposal to confuse @trusted/@safe with requiring @system within @trusted it not a language issue. It is a process issue and can be done with lint-like tooling. Keep @trusted/@safe/@system simple. Enough convoluted semantics in D already.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 8:43 PM, Andrei Alexandrescu wrote: On 2/6/15 3:21 PM, weaselcat wrote: On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote: No, at least three of us, Steven, H.S. Teoh and myself have confirmed that we've moved beyond requesting @trusted blocks. We are no longer requesting them. We are requesting *@system* blocks, which can only appear in @trusted and @system functions. Any unsafe code appearing in a @trusted function which is not inside a @system block is an error. We've changed the name, but I think it will make a world of difference regarding how you will look at it. Marking '@system' code inside a @trusted function is exactly what is requested. Your message about '@trusted' being only acceptable as an interface has been heard. There will be no @trusted blocks, only @system blocks, which are the exact same thing, except they can only appear in @trusted and @system functions. This solution appeals to me greatly. It pinpoints precisely where unsafe code can generate; it catches unintended safety violations in all @trusted code outside @system blocks, as requested by many people so far; it makes systems programming highly visible, with redundancy at the function signature and at the unsafe code itself. I really think it's spot on! this sounds interesting, is anyone going to make a DIP for it? Consider the previous code: https://github.com/D-Programming-Language/phobos/blob/accb351b96bb04a6890bb7df018749337e55eccc/std/file.d#L194 that got replaced with: https://github.com/D-Programming-Language/phobos/blob/master/std/file.d#L194 With the system proposal we're looking at something like: Please understand, Nobody is saying let's replace incorrect code with the same incorrect code with different tags! The idea is to properly identify which code needs more scrutiny, and keep the mechanical checking of @safe where we can. -Steve
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 11:32:41 UTC, Steven Schveighoffer wrote: The idea is that @trusted code still has to be reviewed for memory issues, but is mechanically checked for most of the function for obvious @safe violations. It limits to a degree the scrutiny one must apply to the @trusted function. Remember, the whole point of a @trusted function is that it's manually verified. This is the wrong way to do it and this is a tooling issue, not a language issue. The right way to do it is this: 1. annotate the @trusted region manually where it is needed 2. mechanically verify the whole @trusted region Of course, then you also need a theorem prover... You are trying to do this: 1. mechanically verify the whole @trusted region 2. manually verify the whole @trusted region, but be sloppy about it here an there 3. Ooops, we were sloppy in the wrong spot... Not good.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 4:36 PM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=
ola.fosheim.grostad+dl...@gmail.com wrote:
On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer wrote:
In the proposal, @trusted code is actually considered the same as
@safe, but allows @system escapes.
But that can't work:
@trusted_is_safe {
auto tmp = get_hardware_config();
@system{
mess_up_hardware_config();
}
// now this unsafe call is called in a @safe context, but is unsafe...
// DMD does not catch this, so @trusted_is_safe is broken
call_safe_code_that_now_is_messed_up();
@system{
restore_hardware_config(tmp);
}
}
The idea is that @trusted code still has to be reviewed for memory
issues, but is mechanically checked for most of the function for obvious
@safe violations. It limits to a degree the scrutiny one must apply to
the @trusted function.
Remember, the whole point of a @trusted function is that it's manually
verified.
-Steve
Re: @trust is an encapsulation method, not an escape
On 2/6/15 5:19 PM, Meta wrote: On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer wrote: In the proposal, @trusted code is actually considered the same as @safe, but allows @system escapes. That seems like a good idea and in the spirit of what the goal is. However, won't it be a breaking change? Yes. The big question is, is it worth it? I would say yes, since @trusted is already incorrectly used in most cases. -Steve
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 06:20:16 UTC, Walter Bright wrote: On 2/6/2015 9:49 PM, Vladimir Panteleev wrote: On Friday, 6 February 2015 at 21:08:21 UTC, Walter Bright wrote: 1. exceptions are not for debugging the logic of your program 2. do not use exceptions to recover from logic bugs in your program OK, this is nothing new, but still doesn't answer my question. You wrote is clearly a program bug. Therefore use assert(). It's as simple as that. OK, thank you. A few years ago you were recommending something else for this situation. We were in disagreement then, and I agree with your current opinion.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote: On 2/6/2015 3:02 PM, Zach the Mystic wrote: This solution appeals to me greatly. It pinpoints precisely where unsafe code can generate; it catches unintended safety violations in all @trusted code outside @system blocks, as requested by many people so far; it makes systems programming highly visible, with redundancy at the function signature and at the unsafe code itself. I really think it's spot on! I suspect that such a feature would simply lull people into a false sense of security in that merely tagging an unsafe cast with @system and the compiler accepting it is good enough. My evidence for this is how @trusted was used in Phobos. I'm also not saying phobos was written perfectly to begin with. I think that this whole @system blocks suggestion came up in a slightly different context than did your original discontent with std.file and std.array. I'm not sure you're ever going to stop careless programmers from getting their bad code to compile. But I think that's a different issue from giving good, careful programmers the tools they need. Right now, @trusted functions are the only tool they have to help people figure out where the unsafe code is, but it's not enough. Nested @trusted functions, and @trusted lambas are clearly being added as a workaround for not being able to un@trust large portions of code intended to be @safe.
Re: @trust is an encapsulation method, not an escape
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:
The solution is to regard @trusted as a means of encapsulating
unsafe operations, not escaping them. Encapsulating them means
that the interface from the @trusted code is such that it is
usable from safe code without having to manually review the
safe code for memory safety. For example (also from std.array):
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
I don't have to review callers of trustedMemory() because it
encapsulates an unsafe operation (memcpy) with a safe interface.
If I understand correctly, your rule o be a trusted function is:
Unable to create a memory corrutpion whatever the arguments.
But here:
- dest or src could be the null slice
- the assert would go away in release
So I though this example _could_ corrupt memory?
The reason @trusted applies only to functions, and not to
blocks of code, is that functions are the construct in D that
provides an interface. Arbitrary blocks of code do not have a
structured interface. Adding @trusted { code } support will
encourage incorrect uses like the opening example. The
existence of @trusted blocks will require review of every line
of code in the function that encloses it, and transitively
every function that calls it!
Adding @trusted as a function attribute, on the other hand,
only requires review of the function's interface to determine
if it is acceptable to use in safe code. Safety review of its
callers is unnecessary.
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 10:02:23 UTC, ponce wrote: If I understand correctly, your rule o be a trusted function is: Unable to create a memory corrutpion whatever the arguments. But here: - dest or src could be the null slice - the assert would go away in release So I though this example _could_ corrupt memory? I see know it was already addressed: http://forum.dlang.org/post/mb1vmt$etn$1...@digitalmars.com
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 18:52:45 UTC, Andrei Alexandrescu wrote: I think the problem is overstated. -- Andrei I think there could hardly be a more persuasive argument that this belief is wrong than Walter himself just having made the mistake several times, and not even immediately realizing what is wrong: https://github.com/D-Programming-Language/phobos/pull/2966 [1] Sorry for singling out this one example here. While it is particularly demonstrative, I am certainly not intending to put Walter in a bad light. It's simply hard to get that stuff right, as templates can make it hard to accurately determine the complete public interface of a function. As pointed out in the PR, here are some more examples for this class of bugs from Phobos code, also written and reviewed by top D coders: https://issues.dlang.org/show_bug.cgi?id=14135 https://issues.dlang.org/show_bug.cgi?id=14136 https://issues.dlang.org/show_bug.cgi?id=14137 https://issues.dlang.org/show_bug.cgi?id=14138 Neither of those issues would have been prevented by your new guidelines; the code in question is already written in that way. Quite to the contrary, consequent application of minimal @trusted blocks or even the workaround you reject so strongly would have prevented all of the bugs except for 14138. David [1] The diff in question, for when the PR is fixed: https://github.com/klickverbot/phobos/commit/db647f62cb5279ae42ad98665cd60cdcdb9b3dd5
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 12:40:26 UTC, David Nadlinger wrote: Neither of those issues would have been prevented by your new guidelines; the code in question is already written in that way. Quite to the contrary, consequent application of minimal @trusted blocks or even the workaround you reject so strongly would have prevented all of the bugs except for 14138. This is an incredibly poor argument. The fact that there is no documentation for why the functions are @trusted and why they have to be @trusted is testament to a flawed process. If you insist on relying on half-assed flawed verification you only catch those instances where it should not have been @trusted in the first place, and which would have been caught at an early stage with a decent process, but you will keep missing out on the hard to detect cases. You will run into much more difficult problems if you don't do something about the safety review process. Fix the weak typing rather than making the language more convoluted, the latter compounds the problem in the long run.
Re: @trust is an encapsulation method, not an escape
On 2/7/15 4:40 AM, David Nadlinger wrote: On Friday, 6 February 2015 at 18:52:45 UTC, Andrei Alexandrescu wrote: I think the problem is overstated. -- Andrei I think there could hardly be a more persuasive argument that this belief is wrong than Walter himself just having made the mistake several times, and not even immediately realizing what is wrong: https://github.com/D-Programming-Language/phobos/pull/2966 [1] Sorry for singling out this one example here. While it is particularly demonstrative, I am certainly not intending to put Walter in a bad light. It's simply hard to get that stuff right, as templates can make it hard to accurately determine the complete public interface of a function. As pointed out in the PR, here are some more examples for this class of bugs from Phobos code, also written and reviewed by top D coders: https://issues.dlang.org/show_bug.cgi?id=14135 https://issues.dlang.org/show_bug.cgi?id=14136 https://issues.dlang.org/show_bug.cgi?id=14137 https://issues.dlang.org/show_bug.cgi?id=14138 Neither of those issues would have been prevented by your new guidelines; the code in question is already written in that way. Quite to the contrary, consequent application of minimal @trusted blocks or even the workaround you reject so strongly would have prevented all of the bugs except for 14138. David [1] The diff in question, for when the PR is fixed: https://github.com/klickverbot/phobos/commit/db647f62cb5279ae42ad98665cd60cdcdb9b3dd5 Nice, thanks for this work. One good guideline here is to almost always let generic code rely on deduction instead of ascribing safety attributes to it. Andrei
Re: @trust is an encapsulation method, not an escape
https://github.com/klickverbot/phobos/commit/db647f62cb5279ae42ad98665cd60cdcdb9b3dd5 Nice, thanks for this work. One good guideline here is to almost always let generic code rely on deduction instead of ascribing safety attributes to it. Andrei And making this work in functions that already marked @trusted and need to be, but should have some parts inferred is the hole point of this thread and the @trusted-misuse in phobos.
Re: @trust is an encapsulation method, not an escape
And I'll add this: Please do not turn the compiler into a inadequate verification tool. The compiler should do what it can do well, but what it cannot do it should not do at all and leave to an external verification tool. @trusted basically tells the compiler this is beyond your capability so we left it to someone else. Third parties should be able to provide incrementally improved verification tools, for various purposes, without mandating language or compiler changes. So what you want from the language is simple clean semantics and reasonable best practice annotations for verification that can be extended based on the needs in a particular environment.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 23:21:50 UTC, weaselcat wrote: On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote: No, at least three of us, Steven, H.S. Teoh and myself have confirmed that we've moved beyond requesting @trusted blocks. We are no longer requesting them. We are requesting *@system* blocks, which can only appear in @trusted and @system functions. Any unsafe code appearing in a @trusted function which is not inside a @system block is an error. We've changed the name, but I think it will make a world of difference regarding how you will look at it. Marking '@system' code inside a @trusted function is exactly what is requested. Your message about '@trusted' being only acceptable as an interface has been heard. There will be no @trusted blocks, only @system blocks, which are the exact same thing, except they can only appear in @trusted and @system functions. This solution appeals to me greatly. It pinpoints precisely where unsafe code can generate; it catches unintended safety violations in all @trusted code outside @system blocks, as requested by many people so far; it makes systems programming highly visible, with redundancy at the function signature and at the unsafe code itself. I really think it's spot on! this sounds interesting, is anyone going to make a DIP for it? It was Teoh's idea. Maybe he should have the honors?
Re: @trust is an encapsulation method, not an escape
On 2/6/15 11:29 AM, Zach the Mystic wrote: The best evidence I have it that everyone who actually worked on the phobos code you disapprove of says it's a problem. I see more like two. -- Andrei
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 17:50:05 UTC, Tobias Pankrath wrote:
I was referring to a hypothetical untrusted block that might
be used something like this:
---
void foo(Range)(Range r) @trusted {
// ...
untrusted {
r.front;
}
// Your manually checked code.
untrusted {
r.popFront;
}
// …
}
---
Using current semantics we must not mark foo @trusted, if
r.front and
r.popFront aren't. Using the proposed @safe-blocks (are those
untrusted blocks the same?) we could guarantee that, by
wrapping the use of r.front and r.popFront in @safe-blocks.
This is limiting because now we cannot provide an foo marked
@system for all @system ranges without boiler plate or
duplicating the function.
Correct?
Yes.
The untrusted blocks were an ad-hoc invention to show how @safe
blocks could be modified to actually work in this case (for
template functions, don't require @safe, but mark the function
@system if the contents are unsafe), but that this modification
results in a much less desirable design than just straight up
having @trusted blocks.
David
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 3:57 AM, Martin Krejcirik wrote:
If I understand it correctly, Walter is against adding trusted blocks (trusted
{...}) into @safe functions. But what about having safe blocks in @trusted
functions ?
int somefunc() @trusted
{
int a = systemfunc();
int b;
@safe {
b = safefunc(); // calling systemfunc() not allowed;
}
return a + b;
}
The problem is, again:
@trusted code must have a safe interface
and with code blocks, there is no interface specified to them other than
examining EVERY line of code in it.
The way interfaces are specified in D is to use functions.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 12:36 PM, Atila Neves wrote:
I'm trying to promote suggesting '@system' blocks instead of
'@trusted'. '@trusted' functions, but '@system' blocks - which can
only go in @trusted functions (@system block in @system functions are
redundant). It's the same semantics, but it might win the day because
the intent is to isolate the @system code, while still presenting a
@trusted interface, as seems so important to the leadership.
That might be better than using @safe inside @trusted:
@trusted void myfunc() {
//implicitly safe
@system { //wouldn't compile otherwise.
auto ptr = cast(ubyte*)4;
}
//implicitly safe again
}
BTW, this is H.S. Teoh's suggestion too, and I like it better than mine
after much thought.
-Steve
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 17:13:09 UTC, Zach the Mystic wrote: It's been suggested that '@system' be used to mark the blocks in question. The point would be to emphasize the dangerousness of the operation. The function is still @trusted, but inside, the @system code is marked as such. This is precisely equivalent to just making the function @safe and using @trusted blocks inside (which is incidentally my preferred approach, and similar to what parts of Phobos try to emulate). As I argued almost three years ago in my @trusted considered harmful post, @trusted doesn't differ in meaning from @safe for API clients. Both mean that you can call the function from @safe code, nothing more, nothing less. I hope we agree on that. You might argue that having @trusted in the function declaration serves as an indicator that the implementation requires extra careful review. However, there will *still* be a @trusted keyword sitting in the source code and staring at your face even with @trusted blocks. And if you don't have the source code for a function you call to begin with, all bets are off anyway (i.e. you need to trust its author) as @safe functions may always internally call @trusted ones. If the leadership feels that your proposal is to be preferred because it keeps @trusted in the function signature (as he correctly remarked, the safety of @trusted blocks might depend on surrounding @safe code), then so be it. However, it makes an implementation detail part of the API (see @trusted considered harmful), and furthermore is not backwards-compatible. David
Re: @trust is an encapsulation method, not an escape
On 2/6/15 2:16 PM, Andrei Alexandrescu wrote: On 2/6/15 11:11 AM, H. S. Teoh via Digitalmars-d wrote: On Fri, Feb 06, 2015 at 10:52:45AM -0800, Andrei Alexandrescu via Digitalmars-d wrote: On 2/6/15 10:42 AM, David Nadlinger wrote: On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote: It's clear. I just don't think it's a good point. -- Andrei I'm not making a point; I'm posing a problem. What is your solution? I think the problem is overstated. -- Andrei This is precisely why I have lost all interest in @safe. It's clear that the present problematic situation will continue to hold, and the decision makers are not interested to address it. I am not going to waste any more time and energy on this topic. I've asked repeatedly for evidence of the problematic situation, and all I got was doomsday predictions maintenance nightmare!. If you have such, please show it. If not, thanks for a good course of action. -- Andrei I think your strawman is overstated. The doomsday is the current situation to which you and Walter have objected. If you think having better discipline in reviews is going to fix it, I guess we will have to wait and see what the evidence eventually does show. There isn't evidence that either solution has worked, because neither has been employed yet. Logically, it makes sense to me that we should adjust how @trusted operates to prevent preventable problems that you have identified. But we can just keep the status quo and rely on manual process improvements instead. It's not terribly important to fix it right now, we can try your way first, I don't see how adjusting the meaning of @trusted in the future would be any more disruptive than it would be now. If this is how it is to be, can we get some guidelines as to what should and should not pass review for @trusted? -Steve
Re: @trust is an encapsulation method, not an escape
On Fri, Feb 06, 2015 at 10:52:45AM -0800, Andrei Alexandrescu via Digitalmars-d wrote: On 2/6/15 10:42 AM, David Nadlinger wrote: On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote: It's clear. I just don't think it's a good point. -- Andrei I'm not making a point; I'm posing a problem. What is your solution? I think the problem is overstated. -- Andrei This is precisely why I have lost all interest in @safe. It's clear that the present problematic situation will continue to hold, and the decision makers are not interested to address it. I am not going to waste any more time and energy on this topic. T -- Question authority. Don't ask why, just do it.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 3:02 PM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= ola.fosheim.grostad+dl...@gmail.com wrote: On Friday, 6 February 2015 at 18:51:34 UTC, Steven Schveighoffer wrote: I see the point now that making sure @safe functions don't have escapes has the advantage of not requiring *as much* review as a @system or @trusted function. I am leaning so much towards H.S. Teoh's solution of making @trusted safe by default, and allowing escapes into @system code. That seems like the right abstraction. Just to make sure that I got this right: I don't really understand why you need to escape to @system from @trusted. Isn't @trusted the same as @system but with a seal that says that it has been manually verified to be memory safe? @system simply allows the same internal semantics as @trusted but with no such declared guarantee to the caller? In the proposal, @trusted code is actually considered the same as @safe, but allows @system escapes. I don't have any time to read your further points, but I will catch up with them later, sorry! -Steve
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 4:17 AM, Kagamin wrote:
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:
On 2/6/2015 12:31 AM, Kagamin wrote:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Should be enforce: assert doesn't guard against malicious usage.
Cue my endless attempts to explain the difference between input errors and
logic errors :-(
A little offtop: if this function is compiled in release mode and compiler
assumes assert holds, it's free to use dest.length instead of src.length and if
at runtime dest is longer than src, this will create heartbleed-like bug in safe
code.
Sigh. Please visit your nearest Catholic school and ask one of the nuns to
thwack your knuckles with a ruler!
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 18:58:27 UTC, David Nadlinger wrote: On Friday, 6 February 2015 at 17:13:09 UTC, Zach the Mystic wrote: It's been suggested that '@system' be used to mark the blocks in question. The point would be to emphasize the dangerousness of the operation. The function is still @trusted, but inside, the @system code is marked as such. This is precisely equivalent to just making the function @safe and using @trusted blocks inside (which is incidentally my preferred approach, and similar to what parts of Phobos try to emulate). No it's not. Forcing @trusted functions, with @system blocks inside builds a redundancy into the system. Now someone from outside the can read the signature and know there might be a problem, and someone inside can find everywhere that problem could possibly have arisen from. The difference is you now have two signals instead of one - one in the function header, and the other(s) in the code itself. As I argued almost three years ago in my @trusted considered harmful post, @trusted doesn't differ in meaning from @safe for API clients. Both mean that you can call the function from @safe code, nothing more, nothing less. I hope we agree on that. I know. This was recently pointed out again by Steven Schveighoffer, and it's a great insight. You might argue that having @trusted in the function declaration serves as an indicator that the implementation requires extra careful review. However, there will *still* be a @trusted keyword sitting in the source code and staring at your face even with @trusted blocks. And if you don't have the source code for a function you call to begin with, all bets are off anyway (i.e. you need to trust its author) as @safe functions may always internally call @trusted ones. It is a redundancy, yes, but one people might actually *appreciate*. People around here take memory safety very seriously, and probably with good reason! If the leadership feels that your proposal is to be preferred because it keeps @trusted in the function signature (as he correctly remarked, the safety of @trusted blocks might depend on surrounding @safe code), then so be it. However, it makes an implementation detail part of the API (see @trusted considered harmful), and furthermore is not backwards-compatible. It's not backward-compatible, but it's not hard to fix at all. You get an error, and you enclose your unsafe code in a @system block. And the quality of your code improves at the same time. The only reason the implementation detail, @trusted, was made part of the API was precisely because memory safety is taken extremely seriously by the language designers. There's no other reason I can think of, but for some reason, I can appreciate the thought. Also, I want to find a real solution and I doubt anyone is suggesting getting rid of @trusted at this point.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 11:53 AM, Steven Schveighoffer wrote: I think your strawman is overstated. The doomsday is the current situation to which you and Walter have objected. I see that just as: code in poor style made its way in Phobos. It doesn't improve anything (e.g. didn't find bugs in std.file.read) and is just a net negative resulting from the corrupted use of a feature. There's no completely automated protection against poor style. If you think having better discipline in reviews is going to fix it, I guess we will have to wait and see what the evidence eventually does show. There isn't evidence that either solution has worked, because neither has been employed yet. Logically, it makes sense to me that we should adjust how @trusted operates to prevent preventable problems that you have identified. But we can just keep the status quo and rely on manual process improvements instead. It's not terribly important to fix it right now, we can try your way first, I don't see how adjusting the meaning of @trusted in the future would be any more disruptive than it would be now. If this is how it is to be, can we get some guidelines as to what should and should not pass review for @trusted? Wise words. Walter has a PR on docs, you may want to review it: https://github.com/D-Programming-Language/dlang.org/pull/890 Andrei
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 5:13 AM, Vladimir Panteleev wrote: So, to repeat my question: which one is it? Have you changed your mind, or are there exceptions to the rules in the post you quoted? It means that you, the programmer, have to decide whether it is environmental input validation or a logic error in your code. Follow the rules: 1. exceptions are not for debugging the logic of your program 2. do not use exceptions to recover from logic bugs in your program I have pontificated on this at GREAT length in multiple threads in this n.g. If it is still a mystery to you or anyone else, I give up. Keep in mind the levels of expertise: newbie: follow the rules because you're told they're the right thing to do master: follow the rules because you understand they're the right thing to do guru: transcend the rules because you know when they don't apply
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 17:12:40 UTC, David Nadlinger wrote: It seems obvious that explicitly whitelisting a small number of potentially dangerous but safe operations is much less error-prone approach than disabling compiler checks for everything and then having to remember to blacklist all unverified external dependencies. David That seems obvious to me too. Isn't the whole purpose of having '@trusted' in the first place to direct a programmer who's having memory safety problems to the potential sources those problems? But why have this and then stop at the function level? Why not force the programmer to tag precisely those portions of his code which cause him to tag his function @trusted to begin with? Why help him get to the function, and then leave him hanging out to dry once inside the function?
Re: @trust is an encapsulation method, not an escape
On 2/6/15 10:36 AM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= ola.fosheim.grostad+dl...@gmail.com wrote: On Friday, 6 February 2015 at 15:10:18 UTC, Steven Schveighoffer wrote: into suspect the whole function. So marking a function @safe, and having it mean this function has NO TRUSTED OR SYSTEM CODE in it whatsoever, is probably the right move, regardless of any other changes. But that would break if you want to call a @safe function with a @trusted function reference as a parameter? Or did I misunderstand what you meant here? The whole point of marking a function trusted instead of a block is that you have to follow the rules of function calling to get into that block, and your separate function only has access to variables you give it. My point was that if you have @trusted escapes inside a function, whether it's marked @safe or not, you still have to review the whole function. If the compiler disallowed this outright, then you don't have that issue. Separating the trusted code from the safe code via an API barrier has merits when it comes to code review. Now, @trusted static nested functions that stand on their own are fine, they are no different than public ones, just not public. @trusted static nested functions that are ONLY OK when called in certain ways, that is where we run into issues. At that point, you have to make a choice -- add (somewhat unnecessary) machinery to make sure the function is always called in a @safe way, or expand the scope of the @trusted portion, possibly even to the whole @safe function. I see the point now that making sure @safe functions don't have escapes has the advantage of not requiring *as much* review as a @system or @trusted function. I am leaning so much towards H.S. Teoh's solution of making @trusted safe by default, and allowing escapes into @system code. That seems like the right abstraction. And... what happens if you bring in a new architecture that requires @trusted implementation of a library function that is @safe on other architectures? Then you create a @trusted wrapper around that API, ensuring when called from @safe code it can't corrupt memory. 1. A way to say this function needs extra scrutiny 2. Mechanical verification as MUCH AS POSSIBLE, and especially for changes to said function. Yes, we can do 2 manually if necessary. But having a compiler that never misses on pointing out certain bad things is so much better than not having it. I am not sure if it is worth the trouble. If you are gonna conduct a semi formal proof, then you should not have a mechanical sleeping pillow that makes you sloppy. ;-) I see what you mean, but there are also really dumb things that people miss that a compiler won't. Having a mechanical set of eyes in addition to human eyes is still more eyes ;) Also if you do safety reviews they should be separate from the functional review and only focus on safety. Maybe it would be interesting to have an annotation for @notprovenyet, so that you could have regular reviews during development and then scan the source code for @trusted functions that need a safety review before you a release is permitted? That way you don't have to do the safety review for every single mutation of the @trusted function. The way reviews are done isn't anything the language can require. Certainly we can provide guidelines, and we can require such review processes for phobos and druntime. -Steve
Re: @trust is an encapsulation method, not an escape
On 2/6/15 10:42 AM, David Nadlinger wrote: On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote: It's clear. I just don't think it's a good point. -- Andrei I'm not making a point; I'm posing a problem. What is your solution? I think the problem is overstated. -- Andrei
Re: @trust is an encapsulation method, not an escape
On 2/6/15 11:11 AM, H. S. Teoh via Digitalmars-d wrote: On Fri, Feb 06, 2015 at 10:52:45AM -0800, Andrei Alexandrescu via Digitalmars-d wrote: On 2/6/15 10:42 AM, David Nadlinger wrote: On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote: It's clear. I just don't think it's a good point. -- Andrei I'm not making a point; I'm posing a problem. What is your solution? I think the problem is overstated. -- Andrei This is precisely why I have lost all interest in @safe. It's clear that the present problematic situation will continue to hold, and the decision makers are not interested to address it. I am not going to waste any more time and energy on this topic. I've asked repeatedly for evidence of the problematic situation, and all I got was doomsday predictions maintenance nightmare!. If you have such, please show it. If not, thanks for a good course of action. -- Andrei
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 19:16:13 UTC, Andrei Alexandrescu wrote: This is precisely why I have lost all interest in @safe. It's clear that the present problematic situation will continue to hold, and the decision makers are not interested to address it. I am not going to waste any more time and energy on this topic. I've asked repeatedly for evidence of the problematic situation, and all I got was doomsday predictions maintenance nightmare!. If you have such, please show it. If not, thanks for a good course of action. -- Andrei The best evidence I have it that everyone who actually worked on the phobos code you disapprove of says it's a problem. Argumentum ad populem, I know, but it calls for close scrutiny, to say the least. My attitude is not based on evidence. It's based on just thinking about the problem: http://forum.dlang.org/post/eeglnychgudcffpjc...@forum.dlang.org I really can't answer this question.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 18:51:34 UTC, Steven Schveighoffer wrote: My point was that if you have @trusted escapes inside a function, whether it's marked @safe or not, you still have to review the whole function. If the compiler disallowed this outright, then you don't have that issue. Ok, I also prefer that @trusted only apply to whole functions. I don't agree with the argument, but let's leave it at that ;-). Separating the trusted code from the safe code via an API barrier has merits when it comes to code review. Yes. And I actually don't think @trusted functions in Phobos should contain version() or other forms for conditional compilation since that makes it much harder to reason about. I see the point now that making sure @safe functions don't have escapes has the advantage of not requiring *as much* review as a @system or @trusted function. I am leaning so much towards H.S. Teoh's solution of making @trusted safe by default, and allowing escapes into @system code. That seems like the right abstraction. Just to make sure that I got this right: I don't really understand why you need to escape to @system from @trusted. Isn't @trusted the same as @system but with a seal that says that it has been manually verified to be memory safe? @system simply allows the same internal semantics as @trusted but with no such declared guarantee to the caller? Except that in @system you could potentially switch the stacks and do other unsafe operations that are not brought back to normal before leaving the system context... In @trusted you are required to restore the context to normal before returning. So in the type system you only have @safe and @system, @trusted is just @safe with flexible manual verification rather than the limited automated verification DMD is capable of. Thus you only need to export @safe vs @system for separate compilation...? Isn't that how it is supposed to work already? I see what you mean, but there are also really dumb things that people miss that a compiler won't. Having a mechanical set of eyes in addition to human eyes is still more eyes ;) Well, if you can come up with something sound. The way I see it, @trusted functions are allowed to create a new context on entry as long as it restores the previous context before exiting. This essentially means that what is @safe before entering @trusted no longer can be guaranteed to be safe. The floating point unit might work differently and result in memory unsafe operations etc etc. So calling @safe from @trusted means that you are calling @safe as if it was @system. And therefore @safe code called from @trusted has to be proven correct just like @system code called from @trusted. The way reviews are done isn't anything the language can require. Certainly we can provide guidelines, and we can require such review processes for phobos and druntime. Yes, in Phobos you need to impose additional stricter guarantees in order to support the formal review process and ensure that the formal review cannot be broken without a new review taking place. And yes, that is a process requirement, not a language requirement. A different process might impose other requirements... I think you will have to define such a process though, because I don't think there is a solution for fully automated verification for D without going for a much more complex type system and mechanics (which I actually think is out of reach unless everything is designed around it). Fortunately with 3 reviewers you can do quite well if the proof is available and they work independently without sharing results before everyone is done. If each reviewer has a 10% chance of failure then you end up with only 0.1% chance of all of them failing... So it is possible to get decent results with a formal process in place. Complicated @trusted code (hard to prove safe) should be rejected and unnecessary @trusted code should be fixed in the compiler optimizer or by adding language features (like SIMD).
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer
wrote:
In the proposal, @trusted code is actually considered the same
as @safe, but allows @system escapes.
But that can't work:
@trusted_is_safe {
auto tmp = get_hardware_config();
@system{
mess_up_hardware_config();
}
// now this unsafe call is called in a @safe context, but is
unsafe...
// DMD does not catch this, so @trusted_is_safe is broken
call_safe_code_that_now_is_messed_up();
@system{
restore_hardware_config(tmp);
}
}
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 7:48 AM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= ola.fosheim.grostad+dl...@gmail.com wrote: Then how is this more work than implementing something like a linear type system that is both tedious for the programmer and has taken Rust 8 years to get into working shape...? Rust has unsafe blocks with specific instructions that it cannot be verified mechanically and it is up to the programmer to ensure a safe interface to it. So no, Rust didn't get that working, either, and it is far beyond current compiler technology to do it.
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 7:14 AM, Wyatt wrote: The current @trusted semantics (and accompanying politics) make it exceedingly clear that @safe is meaningless for anything beyond trivial, one-off tools that will never receive maintenance. You are correct in how @trusted is currently (mis)used in Phobos. We aim to fix this. @trusted must provide a safe interface. No exceptions. Yes, that requires review of @trusted code by someone who thoroughly understands this, but there's no other way.
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 11:11 AM, H. S. Teoh via Digitalmars-d wrote: This is precisely why I have lost all interest in @safe. It's clear that the present problematic situation will continue to hold, and the decision makers are not interested to address it. I am not going to waste any more time and energy on this topic. In this thread at 8:20PM last night, Dicebot asked me: I am not even sure how you can show the example though, to be honest - implied issues are about maintaining code and not just writing it. And I replied with a specific example of how to fix one aspect of std.array. There have been no replies. What else can I do to address it?
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 22:24:48 UTC, Walter Bright wrote: Rust has unsafe blocks with specific instructions that it cannot be verified mechanically and it is up to the programmer to ensure a safe interface to it. So no, Rust didn't get that working, either, and it is far beyond current compiler technology to do it. Rust guarantees, though, that all code outside of unsafe blocks/functions is completely safe, which D doesn't do because of trusted. I think that `unsafe` in Rust is more like @trust in D, but I'm not completely sure about that.
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 2:39 PM, Meta wrote: On Friday, 6 February 2015 at 22:24:48 UTC, Walter Bright wrote: Rust has unsafe blocks with specific instructions that it cannot be verified mechanically and it is up to the programmer to ensure a safe interface to it. So no, Rust didn't get that working, either, and it is far beyond current compiler technology to do it. Rust guarantees, though, that all code outside of unsafe blocks/functions is completely safe, which D doesn't do because of trusted. I think that `unsafe` in Rust is more like @trust in D, but I'm not completely sure about that. Rust guarantees no such thing, because it is explicitly up to the Rust programmer to verify a safe interface to unsafe code blocks. It is no different from the D @trusted programmer verifying a safe interface to @trusted code.
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 3:02 PM, Zach the Mystic wrote: No, at least three of us, Steven, H.S. Teoh and myself have confirmed that we've moved beyond requesting @trusted blocks. We are no longer requesting them. We are requesting *@system* blocks, which can only appear in @trusted and @system functions. Any unsafe code appearing in a @trusted function which is not inside a @system block is an error. We've changed the name, but I think it will make a world of difference regarding how you will look at it. Marking '@system' code inside a @trusted function is exactly what is requested. Your message about '@trusted' being only acceptable as an interface has been heard. There will be no @trusted blocks, only @system blocks, which are the exact same thing, except they can only appear in @trusted and @system functions. This solution appeals to me greatly. It pinpoints precisely where unsafe code can generate; it catches unintended safety violations in all @trusted code outside @system blocks, as requested by many people so far; it makes systems programming highly visible, with redundancy at the function signature and at the unsafe code itself. I really think it's spot on! I suspect that such a feature would simply lull people into a false sense of security in that merely tagging an unsafe cast with @system and the compiler accepting it is good enough. My evidence for this is how @trusted was used in Phobos.
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 10:58 AM, David Nadlinger wrote: @trusted doesn't differ in meaning from @safe for API clients. Both mean that you can call the function from @safe code, nothing more, nothing less. I hope we agree on that. That is correct. @trusted is a statement about the implementation of a function, not its interface. This suggests that @trusted should apply to blocks of code, not function declarations. Pedantically, I think that would be correct. But as we've seen in usage, this seductively makes for easy incorrect usage of @trusted. Pragmatically, the language should make it harder to write incorrect code. Hence, I view it as the best compromise to make @trusted also apply only at the function level.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote: No, at least three of us, Steven, H.S. Teoh and myself have confirmed that we've moved beyond requesting @trusted blocks. We are no longer requesting them. We are requesting *@system* blocks, which can only appear in @trusted and @system functions. Any unsafe code appearing in a @trusted function which is not inside a @system block is an error. We've changed the name, but I think it will make a world of difference regarding how you will look at it. Marking '@system' code inside a @trusted function is exactly what is requested. Your message about '@trusted' being only acceptable as an interface has been heard. There will be no @trusted blocks, only @system blocks, which are the exact same thing, except they can only appear in @trusted and @system functions. Also, to be clear, any @system block inside a @safe function is an automatic error.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 21:56:40 UTC, Walter Bright wrote:
On 2/6/2015 11:29 AM, Zach the Mystic wrote:
My attitude is not based on evidence. It's based on just
thinking about the
problem:
http://forum.dlang.org/post/eeglnychgudcffpjc...@forum.dlang.org
I really can't answer this question.
You asked:
Why not force the programmer to tag precisely those portions
of his code
which cause him to tag his function @trusted to begin with?
That question has been asked and answered repeatedly. The
answer is that @trusted is not only to TAG unsafe code, but
must provide a SAFE INTERFACE to it.
@trusted {
... unsafe code ...
}
provides no indication of what the interface to the unsafe code
is. Addressing only the TAG aspect is insufficient.
No, at least three of us, Steven, H.S. Teoh and myself have
confirmed that we've moved beyond requesting @trusted blocks. We
are no longer requesting them. We are requesting *@system*
blocks, which can only appear in @trusted and @system functions.
Any unsafe code appearing in a @trusted function which is not
inside a @system block is an error. We've changed the name, but I
think it will make a world of difference regarding how you will
look at it. Marking '@system' code inside a @trusted function is
exactly what is requested. Your message about '@trusted' being
only acceptable as an interface has been heard. There will be no
@trusted blocks, only @system blocks, which are the exact same
thing, except they can only appear in @trusted and @system
functions.
This solution appeals to me greatly. It pinpoints precisely where
unsafe code can generate; it catches unintended safety violations
in all @trusted code outside @system blocks, as requested by many
people so far; it makes systems programming highly visible, with
redundancy at the function signature and at the unsafe code
itself. I really think it's spot on!
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 8:19 AM, John Colvin wrote: Is that what we want? I can't see why not, but it feels off somehow... Effectively you've got @trusted blocks in an @trusted function, just inverted. Right. It just inverted the interface problem, the problem remains. So, no for this idea. Another way to say it: @safe = @trusted must go through a safe interface @trusted = @safe must go through a safe interface
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 11:29 AM, Zach the Mystic wrote:
My attitude is not based on evidence. It's based on just thinking about the
problem:
http://forum.dlang.org/post/eeglnychgudcffpjc...@forum.dlang.org
I really can't answer this question.
You asked:
Why not force the programmer to tag precisely those portions of his code
which cause him to tag his function @trusted to begin with?
That question has been asked and answered repeatedly. The answer is that
@trusted is not only to TAG unsafe code, but must provide a SAFE INTERFACE to it.
@trusted {
... unsafe code ...
}
provides no indication of what the interface to the unsafe code is. Addressing
only the TAG aspect is insufficient.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer wrote: In the proposal, @trusted code is actually considered the same as @safe, but allows @system escapes. That seems like a good idea and in the spirit of what the goal is. However, won't it be a breaking change?
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 5:28 AM, Steven Schveighoffer wrote: It's better because I know where it is used. It's used in one place, and I can squash it right there saying No, you can't do this in this one place. Instead of reviewing an API in ALL POSSBILE CONTEXTS (which if trustedCast is a public API, would be a lot), I have to review one call in ONE CONTEXT. The former is WORSE because it can be used in 100 places. Now I have to go through and fix ALL THOSE FUNCTIONS that use it, because its interface was exposed to the whole of phobos. This is the crux of the problem - failing to define a safe interface to the trusted code block. Without defining an interface, you're right, you must review all the context(s) that call it. With a safe interface you DO NOT. You only have to review the interface. A simple rule: If you need to do a safety review on the context in which @trusted code is called YOU ARE DOING IT WRONG because you've failed to provide a safe interface to the @trusted code. It's like solving a physics problem and winding up with negative energy. If that happens, you made a mistake. It is not a matter of judgement or opinion, it is an objective fact. Going forward, all @trusted code that leaks unsafety into its context will be rejected for inclusion in Phobos. The code reviewer only has to review the @trusted block to determine this - he does not have to review the context.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote: This solution appeals to me greatly. It pinpoints precisely where unsafe code can generate; it catches unintended safety violations in all @trusted code outside @system blocks, as requested by many people so far; it makes systems programming highly visible, with redundancy at the function signature and at the unsafe code itself. I really think it's spot on! Even a call to a @system function inside a @trusted function must occur inside a @system block. It's that strict.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote: I suspect that such a feature would simply lull people into a false sense of security in that merely tagging an unsafe cast with @system and the compiler accepting it is good enough. My evidence for this is how @trusted was used in Phobos. How is adding @system to some operations *in addition to* adding @trusted to the function declaration more likely to lull people into a false sense of security than just adding @trusted right now? Let me also point out that the cases where the @system block equivalent is used right now (like in std.file, or the trustedXyz functions in std.array) are NOT the ones that actually have safety bugs in them (such as std.array.uninitializedArray or std.uuid.randomUUID). The two latter examples were actually written in your preferred style. David
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote: This solution appeals to me greatly. It pinpoints precisely where unsafe code can generate; it catches unintended safety violations in all @trusted code outside @system blocks, as requested by many people so far; it makes systems programming highly visible, with redundancy at the function signature and at the unsafe code itself. I really think it's spot on! I suspect that such a feature would simply lull people into a false sense of security in that merely tagging an unsafe cast with @system and the compiler accepting it is good enough. My evidence for this is how @trusted was used in Phobos. You do realize that our proposal *tightens* security, with no loosening at all? No code which currently fails to compile will start compiling with this proposal. This is literally a breaking change which does nothing but cause errors in existing code - for the explicit purpose of making all code safer, which it will do, possibly dramatically.
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 01:43:01 UTC, Andrei Alexandrescu
wrote:
With the system proposal we're looking at something like:
version (Posix) void[] read(in char[] name, size_t upTo =
size_t.max) @trusted
{
import core.memory;
// A few internal configuration parameters {
enum size_t
minInitialAlloc = 1024 * 4,
maxInitialAlloc = size_t.max / 2,
sizeIncrement = 1024 * 16,
maxSlackMemoryAllowed = 1024;
// }
@system
{
immutable fd =
core.sys.posix.fcntl.open(name.tempCString(),
core.sys.posix.fcntl.O_RDONLY);
}
cenforce(fd != -1, name);
scope(exit) core.sys.posix.unistd.close(fd);
stat_t statbuf = void;
@system
{
cenforce(trustedFstat(fd, trustedRef(statbuf)) == 0,
name);
}
immutable initialAlloc = to!size_t(statbuf.st_size
? min(statbuf.st_size + 1, maxInitialAlloc)
: minInitialAlloc);
void[] result = uninitializedArray!(ubyte[])(initialAlloc);
scope(failure) delete result;
size_t size = 0;
for (;;)
{
@system
{
immutable actual = core.sys.posix.unistd.read(fd,
result.ptr + size),
min(result.length, upTo) - size);
}
cenforce(actual != -1, name);
if (actual == 0) break;
size += actual;
if (size result.length) continue;
immutable newAlloc = size + sizeIncrement;
@system
{
result = GC.realloc(result.ptr, newAlloc,
GC.BlkAttr.NO_SCAN)[0 .. newAlloc];
}
@system
{
return result.length - size = maxSlackMemoryAllowed
? GC.realloc(result.ptr, size,
GC.BlkAttr.NO_SCAN)[0 .. size]
: result[0 .. size];
}
}
We want to move D forward, folks. This is not it.
Andrei
Oh I see. There are three posts, in the latter two of which the
little @trusted functions are already removed. I had thought they
were all identical, but you obviously realized the little
functions should be removed.
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 9:49 PM, Vladimir Panteleev wrote: On Friday, 6 February 2015 at 21:08:21 UTC, Walter Bright wrote: 1. exceptions are not for debugging the logic of your program 2. do not use exceptions to recover from logic bugs in your program OK, this is nothing new, but still doesn't answer my question. You wrote is clearly a program bug. Therefore use assert(). It's as simple as that.
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 01:41:19 UTC, Andrei Alexandrescu
wrote:
Consider the previous code:
[...]
static trustedRead(int fildes, void* buf, size_t nbyte)
@trusted
{
return core.sys.posix.unistd.read(fildes, buf, nbyte);
}
static trustedRealloc(void* p, size_t sz, uint ba = 0,
const TypeInfo ti = null) @trusted
{
return GC.realloc(p, sz, ba, ti);
}
static trustedPtrAdd(void[] buf, size_t s) @trusted
{
return buf.ptr+s;
}
static trustedPtrSlicing(void* ptr, size_t lb, size_t ub)
@trusted
{
return ptr[lb..ub];
}
First of all, these little @trusted functions are made obsolete
by the new system. They should certainly be omitted.
@system
{
immutable fd =
core.sys.posix.fcntl.open(name.tempCString(),
core.sys.posix.fcntl.O_RDONLY);
}
Next, you have to realize that @system blocks are like 'try'
blocks: You don't need brackets if there's only one statement in
them.
Here's how I would rewrite what you have written using the new
method:
version (Posix) void[] read(in char[] name, size_t upTo =
size_t.max)
@trusted
{
import core.memory;
// A few internal configuration parameters {
enum size_t
minInitialAlloc = 1024 * 4,
maxInitialAlloc = size_t.max / 2,
sizeIncrement = 1024 * 16,
maxSlackMemoryAllowed = 1024;
// }
@system immutable fd =
core.sys.posix.fcntl.open(name.tempCString(),
core.sys.posix.fcntl.O_RDONLY);
cenforce(fd != -1, name);
scope(exit) core.sys.posix.unistd.close(fd);
stat_t statbuf = void;
@system cenforce(trustedFstat(fd, trustedRef(statbuf)) == 0,
name);
immutable initialAlloc = to!size_t(statbuf.st_size
? min(statbuf.st_size + 1, maxInitialAlloc)
: minInitialAlloc);
void[] result = uninitializedArray!(ubyte[])(initialAlloc);
scope(failure) delete result;
size_t size = 0;
for (;;)
{
@system immutable actual = core.sys.posix.unistd.read(fd,
result.ptr + size, min(result.length, upTo) -
size);
cenforce(actual != -1, name);
if (actual == 0) break;
size += actual;
if (size result.length) continue;
immutable newAlloc = size + sizeIncrement;
@system result = GC.realloc(
result.ptr, newAlloc, GC.BlkAttr.NO_SCAN)[0 ..
newAlloc];
}
@system return result.length - size = maxSlackMemoryAllowed
? GC.realloc(result.ptr, size,
GC.BlkAttr.NO_SCAN)[0..size]
: result[0 .. size];
}
Note that I just mechanically your @system blocks with the better
form. I didn't arrange for them to be elegant. There's nothing
wrong with encapsulating a @trusted sequence in a @system block
with brackets, to aid future reviewers in identifying subsequent
code thought to be affected by the @system statements. Also, few
functions will have their @system blocks more or less evenly
distributed throughout like the above function does.
The new proposal will never let you add an unsafe operation
without your knowing it. It's definitely the way forward.
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 05:35:51 UTC, Zach the Mystic wrote: Note that I just mechanically your @system blocks with the better form. ...mechanically replaced, I mean, of course.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 21:08:21 UTC, Walter Bright wrote: 1. exceptions are not for debugging the logic of your program 2. do not use exceptions to recover from logic bugs in your program OK, this is nothing new, but still doesn't answer my question. Would you say that the correct thing to do, then, would be an unconditional check that throws an Error? Such as, `if (!...) throw new Error(...)`, or `enforce!Error(...)`? I recall there is a bug report on Bugzilla that Phobos contracts are removed in release (default) builds of the library, when ideally contracts of public functions should stay but inner asserts should be removed in optimized code. That would allow using them for validating parameters without resorting to explicitly-unconditional checks.
Re: @trust is an encapsulation method, not an escape
On Saturday, 7 February 2015 at 05:35:51 UTC, Zach the Mystic wrote: Here's how I would rewrite what you have written using the new method: ... stat_t statbuf = void; @system cenforce(trustedFstat(fd, trustedRef(statbuf)) == 0, name); I didn't rewrite this because I didn't see the trustedXXX functions they referred to, but the basic rewrite would be: @system cenforce(fstat(fd, reff(statbuf) == 0), name); In other words, just copy the @system code directly without going through @trusted.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 21:33:01 UTC, Walter Bright wrote: On 2/6/2015 10:58 AM, David Nadlinger wrote: @trusted doesn't differ in meaning from @safe for API clients. Both mean that you can call the function from @safe code, nothing more, nothing less. I hope we agree on that. That is correct. @trusted is a statement about the implementation of a function, not its interface. This suggests that @trusted should apply to blocks of code, not function declarations. Pedantically, I think that would be correct. But as we've seen in usage, this seductively makes for easy incorrect usage of @trusted. Pragmatically, the language should make it harder to write incorrect code. Hence, I view it as the best compromise to make @trusted also apply only at the function level. Please see this post: http://forum.dlang.org/post/riphxcqazksykafum...@forum.dlang.org
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote: No, at least three of us, Steven, H.S. Teoh and myself have confirmed that we've moved beyond requesting @trusted blocks. We are no longer requesting them. We are requesting *@system* blocks, which can only appear in @trusted and @system functions. Any unsafe code appearing in a @trusted function which is not inside a @system block is an error. We've changed the name, but I think it will make a world of difference regarding how you will look at it. Marking '@system' code inside a @trusted function is exactly what is requested. Your message about '@trusted' being only acceptable as an interface has been heard. There will be no @trusted blocks, only @system blocks, which are the exact same thing, except they can only appear in @trusted and @system functions. This solution appeals to me greatly. It pinpoints precisely where unsafe code can generate; it catches unintended safety violations in all @trusted code outside @system blocks, as requested by many people so far; it makes systems programming highly visible, with redundancy at the function signature and at the unsafe code itself. I really think it's spot on! this sounds interesting, is anyone going to make a DIP for it?
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 3:34 PM, David Nadlinger wrote: On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote: I suspect that such a feature would simply lull people into a false sense of security in that merely tagging an unsafe cast with @system and the compiler accepting it is good enough. My evidence for this is how @trusted was used in Phobos. How is adding @system to some operations *in addition to* adding @trusted to the function declaration more likely to lull people into a false sense of security than just adding @trusted right now? Let me also point out that the cases where the @system block equivalent is used right now (like in std.file, or the trustedXyz functions in std.array) are NOT the ones that actually have safety bugs in them (such as std.array.uninitializedArray or std.uuid.randomUUID). The two latter examples were actually written in your preferred style. I've said that the usage of those functions was not actually buggy, what was wrong about them was that they required review of the surrounding supposedly safe context. I.e. they produced a false sense of safety. I fear the @system blocks, even if only allowed in @trusted functions, would produce a similar illusion of safety. I agree with Andrei in that I do not believe that reviewing a @trusted function, line by line, for safety is necessarily some sort of maintenance nightmare. If it is, then a refactoring should be considered to encapsulate the unsafe code in a smaller, simpler manner. I.e. let's make an effort to use @trusted correctly, and then see where we stand. Scott Meyer's excellent article (a classic): http://www.drdobbs.com/cpp/how-non-member-functions-improve-encapsu/184401197 describes this most eloquently. (Just substitute private members with trusted code.)
Re: @trust is an encapsulation method, not an escape
On Fri, Feb 06, 2015 at 04:04:48PM -0800, Walter Bright via Digitalmars-d wrote: [...] I agree with Andrei in that I do not believe that reviewing a @trusted function, line by line, for safety is necessarily some sort of maintenance nightmare. If it is, then a refactoring should be considered to encapsulate the unsafe code in a smaller, simpler manner. [...] This does not take into the account the fact that a @trusted function may call other, non-@trusted, functions. When one of those other functions changes, the @trusted function necessarily needs to be reviewed again. However, under the current implementation, this is not done because when the other, non-@trusted, function is modified, nobody thinks to re-review the @trusted function. They may not even be in the same module. There is no mechanism in place to raise a warning flag when a @trusted function's dependencies are modified. Thus, the proof of safety of the @trusted function has been invalidated, but trust continues to be conferred upon it. T -- Let's call it an accidental feature. -- Larry Wall
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 4:29 PM, H. S. Teoh via Digitalmars-d wrote: This does not take into the account the fact that a @trusted function may call other, non-@trusted, functions. When one of those other functions changes, the @trusted function necessarily needs to be reviewed again. That's correct. However, under the current implementation, this is not done because when the other, non-@trusted, function is modified, nobody thinks to re-review the @trusted function. They may not even be in the same module. There is no mechanism in place to raise a warning flag when a @trusted function's dependencies are modified. Thus, the proof of safety of the @trusted function has been invalidated, but trust continues to be conferred upon it. When the interface to an @system function is changed, all uses of that function have to be reviewed, whether one thinks of it or not. This is part of the review process. Having @system blocks does not alter that.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 3:21 PM, weaselcat wrote:
On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote:
No, at least three of us, Steven, H.S. Teoh and myself have confirmed
that we've moved beyond requesting @trusted blocks. We are no longer
requesting them. We are requesting *@system* blocks, which can only
appear in @trusted and @system functions. Any unsafe code appearing in
a @trusted function which is not inside a @system block is an error.
We've changed the name, but I think it will make a world of difference
regarding how you will look at it. Marking '@system' code inside a
@trusted function is exactly what is requested. Your message about
'@trusted' being only acceptable as an interface has been heard. There
will be no @trusted blocks, only @system blocks, which are the exact
same thing, except they can only appear in @trusted and @system
functions.
This solution appeals to me greatly. It pinpoints precisely where
unsafe code can generate; it catches unintended safety violations in
all @trusted code outside @system blocks, as requested by many people
so far; it makes systems programming highly visible, with redundancy
at the function signature and at the unsafe code itself. I really
think it's spot on!
this sounds interesting, is anyone going to make a DIP for it?
Consider the previous code:
https://github.com/D-Programming-Language/phobos/blob/accb351b96bb04a6890bb7df018749337e55eccc/std/file.d#L194
that got replaced with:
https://github.com/D-Programming-Language/phobos/blob/master/std/file.d#L194
With the system proposal we're looking at something like:
version (Posix) void[] read(in char[] name, size_t upTo = size_t.max)
@trusted
{
import core.memory;
// A few internal configuration parameters {
enum size_t
minInitialAlloc = 1024 * 4,
maxInitialAlloc = size_t.max / 2,
sizeIncrement = 1024 * 16,
maxSlackMemoryAllowed = 1024;
// }
@system
{
immutable fd = core.sys.posix.fcntl.open(name.tempCString(),
core.sys.posix.fcntl.O_RDONLY);
}
cenforce(fd != -1, name);
scope(exit) core.sys.posix.unistd.close(fd);
stat_t statbuf = void;
@system
{
cenforce(trustedFstat(fd, trustedRef(statbuf)) == 0, name);
}
immutable initialAlloc = to!size_t(statbuf.st_size
? min(statbuf.st_size + 1, maxInitialAlloc)
: minInitialAlloc);
void[] result = uninitializedArray!(ubyte[])(initialAlloc);
scope(failure) delete result;
size_t size = 0;
for (;;)
{
@system
{
immutable actual = core.sys.posix.unistd.read(fd,
result.ptr + size),
min(result.length, upTo) - size);
}
cenforce(actual != -1, name);
if (actual == 0) break;
size += actual;
if (size result.length) continue;
immutable newAlloc = size + sizeIncrement;
@system
{
result = GC.realloc(result.ptr, newAlloc,
GC.BlkAttr.NO_SCAN)[0 .. newAlloc];
}
@system
{
return result.length - size = maxSlackMemoryAllowed
? GC.realloc(result.ptr, size, GC.BlkAttr.NO_SCAN)[0 .. size]
: result[0 .. size];
}
}
We want to move D forward, folks. This is not it.
Andrei
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 12:31 AM, Kagamin wrote:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Should be enforce: assert doesn't guard against malicious usage.
Cue my endless attempts to explain the difference between input errors and logic
errors :-(
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 05:32:48 UTC, Walter Bright wrote: Good point. Then a constraint can be added to the function signature that T is copyable. D's type system is strong enough for that. Yes, if you remember to add it. The ideal would be to have the default constrained to proper value-types. D inherits some of C's weaker typing properties by trying to stay Cish to the programmer. The fact that so many messages has been posted to this thread before someone caught it is testament to the burden put upon reviewers with @trusted. After all, in debate-like threads like this people will look with higher scrutiny at posted code than in an average review...
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 10:28:38 UTC, Walter Bright wrote: On 2/6/2015 1:01 AM, Vladimir Panteleev wrote: On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote: Cue my endless attempts to explain the difference between input errors and logic errors :-( So which one is it? http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html Do you want to say, that safe code can still corrupt memory if it's crafted by a malevolent programmer or if it doesn't perform additional domain-specific logic checks of otherwise memory-safe data?
Re: @trust is an encapsulation method, not an escape
If I understand it correctly, Walter is against adding trusted
blocks (trusted {...}) into @safe functions. But what about
having safe blocks in @trusted functions ?
int somefunc() @trusted
{
int a = systemfunc();
int b;
@safe {
b = safefunc(); // calling systemfunc() not allowed;
}
return a + b;
}
Re: @trust is an encapsulation method, not an escape
This prevents that a @trusted/@safe function becoming @system breaks the @trusted guarantee of called functions. Calling function, forward in the example.
Re: @trust is an encapsulation method, not an escape
On 2/6/2015 1:01 AM, Vladimir Panteleev wrote: On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote: Cue my endless attempts to explain the difference between input errors and logic errors :-( So which one is it? http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:
On 2/6/2015 12:31 AM, Kagamin wrote:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright
wrote:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Should be enforce: assert doesn't guard against malicious
usage.
Cue my endless attempts to explain the difference between input
errors and logic errors :-(
So which one is it?
On one hand, it is clearly a logic error - passing arrays of
different length is clearly a program bug.
On the other hand, this is a library function, and as you said,
we can't know how it's going to be used - so the check has to be
unconditional.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:
On 2/6/2015 12:31 AM, Kagamin wrote:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright
wrote:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Should be enforce: assert doesn't guard against malicious
usage.
Cue my endless attempts to explain the difference between input
errors and logic errors :-(
I agree with you, that this should be an assert, but then it
cannot be @trusted. A @trusted function should provide it's
guarantee for all input that can be crafted in @safe code. This
is why this
void bar()
{
ubyte[] a;
a.ptr = 0; // arbitrary value as long as NOT from an allocator
a.len = 1;
setToZero(a);
}
is not a valid counter-example and that memcpy cannot be
@trusted. It has no safe interface.
Back to topic. Since @trusted must provide a safe interface, I
don't think it makes much sense to attach @trusted to smaller
code blocks than functions. That might not be true for the
opposite: @safe blocks where the compiler turns the safety check
back on.
This prevents that a @trusted/@safe function becoming @system
breaks the @trusted guarantee of called functions.
void bar(void* argument) @trusted { ... }
void forward() @trusted
{
// REVIEW: forwards arguments to bar, which is @trusted, so
we can
// trust forward as well
void* p = malloc(...);
bar(arguments);
free(p);
}
Changing bar to @system is a breaking change and currently a
silent one. This is a maintenance problem I see as well. Solution:
void forward() @trusted
{
void* p = malloc(...);
// safe is bar is safe, so turn compiler checks on again.
@safe { bar(p); }
free(p);
}
Introducing an error in an @trusted function is _not_ an
maintenance horror. While your @safe-ty guarantee is out of the
window, it can be catched my reviewing one function: the one
where the error is introduced in. Making a @trusted function
@system forces us to review all @trusted functions calling the
now @system function.
Re: @trust is an encapsulation method, not an escape
FWIW, and now that I think I understand how @trusted is supposed
to be used, the arguments are valid. But I also:
1. Agree with H S Teoh on the maintainability aspect. Depending
on humans reviewing the code is never going to work out. And even
if it did, subsequent edits might break everything. It's a lot
harder to review a 2-line diff to an existing function for
trustworthiness than the original one when it was first written.
I've lost count of how many code reviews I've approved at work
that introduced subtle crashy bugs. And let's face it, reviewing
a @trusted D function is no different from reviewing C code.
2. Agree with Dicebot that reviewing a 50-line function and
making sure it isn't naughty is quite hard. Not impossible, but
hard to guarantee its safety.
Unfortunately, I have no suggestions on how to make it better.
The @safe block/local function in a @trusted function idea
sounds promising, but I don't know how well it'd work out in
practice.
Atila
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:
Consider the following code excerpted from std.array.join:
static U trustedCast(U, V)(V v) @trusted { return cast(U) v; }
return trustedCast!RetType(result);
This is because the compiler would complain that the following
line would not be @safe:
return cast(RetType)(result);
The rationale is I know it's safe, so I'll create an @trusted
wrapper to eliminate the error. What comes next is that's
cumbersome. How about a better syntax:
trusted {
return cast(RetType)(result);
}
? It's the rationale behind unsafe blocks that appear in
other languages. It seems like a perfectly reasonable request.
The trouble with it is, what if the cast is converting from an
integer to a pointer? That could lead to memory corruption. The
code allows a potentially memory corrupting operation to be
inserted into code that is otherwise @safe.
The only way to deal with it is to then manually review
everything about 'RetType' and 'result' to prove to oneself
that it is not converting random bit patterns into pointers. In
other words, one is manually reviewing @safe code for memory
corruption errors.
This is an abject failure of @safe, @trusted, and @system.
The solution is to regard @trusted as a means of encapsulating
unsafe operations, not escaping them. Encapsulating them means
that the interface from the @trusted code is such that it is
usable from safe code without having to manually review the
safe code for memory safety. For example (also from std.array):
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
I don't have to review callers of trustedMemory() because it
encapsulates an unsafe operation (memcpy) with a safe interface.
The reason @trusted applies only to functions, and not to
blocks of code, is that functions are the construct in D that
provides an interface. Arbitrary blocks of code do not have a
structured interface. Adding @trusted { code } support will
encourage incorrect uses like the opening example. The
existence of @trusted blocks will require review of every line
of code in the function that encloses it, and transitively
every function that calls it!
Adding @trusted as a function attribute, on the other hand,
only requires review of the function's interface to determine
if it is acceptable to use in safe code. Safety review of its
callers is unnecessary.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 14:00:24 UTC, Atila Neves wrote: 1. Agree with H S Teoh on the maintainability aspect. Depending on humans reviewing the code is never going to work out. Of course it can work out. You need a formalized process and a group of people with training in math or comp.sci to do the review. Proving memory safety for a single function in a semi-formal manner is not like proving full correctness. I suggest the following: 1. @trusted should only be used on high priority code (not to gain minor speed ups) 2. @trusted code should always carry the safety-proof as embedded comments 3. @trusted templates should carry the most stringent type constraints (and proof required to weaken them) 4. At least 3 reviewers with knowledge of compsci/math MUST verify the soundness of the proof. 5. Have a list of volunteers that are called in by email to review @trusted code. If only 2% is @trusted then this should work out fine. The alternative is to give up @safe completely or completely change the typesystem.
Re: @trust is an encapsulation method, not an escape
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Should be enforce: assert doesn't guard against malicious usage.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 15:14:14 UTC, Wyatt wrote: On Friday, 6 February 2015 at 13:42:40 UTC, Ola Fosheim Grøstad wrote: cannot modify this without detailed review. This quote from Ola, here? That basically describes my job maintaining big piles of legacy C: the compiler verifies nothing, so every change to the anything in the API of safe functions or anything in their entire call chain must be painstakingly reviewed. A single change generally takes me several days of research, verification, and testing. I fixed about 150 potential memory issues (and several really dumb logic errors) with Clang's static analysis when I first inherited the code; it took weeks. (And now writing new stuff using this safe API is turning up memory safety issues anyway!) So from my perspective, calling this situation completely impractical reveals a stunning gift for understatement. Is this really the best we can do after however many years? Because it blows. The current @trusted semantics (and accompanying politics) make it exceedingly clear that @safe is meaningless for anything beyond trivial, one-off tools that will never receive maintenance. -Wyatt Exactly the reason why I never liked C and C++ only makes it better if everyone on team stays away from Cisms and uses the safer alternatives. Now the C and C++ world finally accepted the use of static analyzers, but I had the displeasure to try to find pointer related issues without one, back in the .com days. As you say, it takes weeks. Regarding D, maybe @trusted should go away and only keep @system (== unsafe in other languages), as it can hardly give any extra security guarantee. However, I don't see how to solve the human review process, as that is an halting problem. -- Paulo
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 15:14:14 UTC, Wyatt wrote: So from my perspective, calling this situation completely impractical reveals a stunning gift for understatement. Is this really the best we can do after however many years? Because it blows. The current @trusted semantics (and accompanying politics) make it exceedingly clear that @safe is meaningless for anything beyond trivial, one-off tools that will never receive maintenance. I don't get this. If: 1. @safe actually works. 2. @trusted sections are written without dependencies 3. @trusted are formally proven safe 4. @trusted functions rarely change 5. @trusted is 0-2% of the codebase Then how is this more work than implementing something like a linear type system that is both tedious for the programmer and has taken Rust 8 years to get into working shape...? Jonathan recently said he would like to volunteer some, and he has mentioned a background with math/proofs. If he is willing to do safety review, then so I am, then so will someone else who feel like refreshing their training in program verification... Use the resources you have. Those resources, we do have, I think. Unused. The resources we obviously don't have is experts on type systems and automated proof and verification engines.
Re: @trust is an encapsulation method, not an escape
Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: On Friday, 6 February 2015 at 13:28:59 UTC, Steven Schveighoffer wrote: The bottom line of my reasoning is that code changes over time, by different people. Context is forgotten. It's much better to have the compiler verify you know what you are doing when working with @trusted than it is to just allow anyone to inject code anywhere. Actually, I think this argument goes against what you are arguing for. Anything within a @trusted section has a big warning sign attached to it that says cannot modify this without detailed review. But the compiler cannot verify that a @safe function with local @trusted blocks are actually safe, so it only buys you a false sense of security. I'd go even further: The compiler could even make optimizations in @safe code based on the assumption that all @trusted function calls expose a safe interface. I suspect this will lead to undefined behavior and very subtle bugs.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 3:57 AM, Martin Krejcirik wrote:
If I understand it correctly, Walter is against adding trusted blocks
(trusted {...}) into @safe functions. But what about having safe blocks
in @trusted functions ?
That would be sensible - perhaps the best step forward following this
long discussion. -- Andrei
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu
wrote:
On 2/6/15 3:57 AM, Martin Krejcirik wrote:
If I understand it correctly, Walter is against adding trusted
blocks
(trusted {...}) into @safe functions. But what about having
safe blocks
in @trusted functions ?
That would be sensible - perhaps the best step forward
following this long discussion. -- Andrei
It feels inelegant, but it might be the best way out of a bad
situation.
I can instantly see this happening:
void foo() @trusted
{
@safe
{
//loads of code
}
//a few lines of system code, only safe due to context in the
@safe blocks
@safe
{
\\loads of code
}
}
Is that what we want? I can't see why not, but it feels off
somehow... Effectively you've got @trusted blocks in an @trusted
function, just inverted.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 5:13 AM, Vladimir Panteleev wrote: On Friday, 6 February 2015 at 10:28:38 UTC, Walter Bright wrote: On 2/6/2015 1:01 AM, Vladimir Panteleev wrote: On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote: Cue my endless attempts to explain the difference between input errors and logic errors :-( So which one is it? http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html That doesn't answer my question. A few years ago, I recall, you were arguing that for functions which are or may be exported to a DLL, thus all Phobos functions, it is impossible to predict how the functions will be used. Thus, you argued, the functions' input has to be validated, even if invalid parameters can only be passed to the function as a result of a program bug, and never user input. So, to repeat my question: which one is it? Have you changed your mind, or are there exceptions to the rules in the post you quoted? Could you all please grant me this wish - let's not get into that vortex again? It renders everyone involved unproductive for days on end. Thanks. -- Andrei
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu
wrote:
On 2/6/15 3:57 AM, Martin Krejcirik wrote:
If I understand it correctly, Walter is against adding trusted
blocks
(trusted {...}) into @safe functions. But what about having
safe blocks
in @trusted functions ?
That would be sensible - perhaps the best step forward
following this long discussion. -- Andrei
This still does not solve the template inference problem though,
unless you make it a non-@trusted block instead of requiring
@safe-ty. And then you'd end up with the—at least to my
eyes—rather absurd situation that a template function that is
marked @trusted might actually end up being @system.
David
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 17:02:44 UTC, Wyatt wrote: On Friday, 6 February 2015 at 15:48:45 UTC, Ola Fosheim Grøstad wrote: 2. @trusted sections are written without dependencies This really won't happen unless statically enforced because humans are involved. 3. @trusted are formally proven safe ...by humans? 4. @trusted functions rarely change Is this so? Data, please. 5. @trusted is 0-2% of the codebase In Phobos, you mean? You've checked? At least I expect the amount of @trusted in phobos to be much higher than in normal user code.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 16:50:51 UTC, Tobias Pankrath wrote:
On Friday, 6 February 2015 at 16:40:10 UTC, David Nadlinger
wrote:
This still does not solve the template inference problem
though, unless you make it a non-@trusted block instead of
requiring @safe-ty.
Don't understand. If that matters, the code shouldn't be marked
@trusted
in the first place.
Let's say you have a template function that accepts a range. For
performance, you want to do some of the processing in a way that
is @system, but can be verified to be correct for all inputs in
this specific case. In other words, that piece of code can be
rightfully @trusted. However, marking the whole function as
@trusted would be a mistake, as the primitives of the range that
is your input data might be @system.
Using @trusted blocks (which is what is currently emulated by the
nested functions/lambdas), you can just mark your unsafe code as
@trusted and let the compiler figure out the safety of the whole
function. @safe blocks wouldn't work for this, as you'd
inadvertently require the user-supplied input range to have
@safe/@trusted primitives.
And then you'd end up with the—at least to my eyes—rather
absurd situation that a template function that is marked
@trusted might actually end up being @system.
David
How?
I was referring to a hypothetical untrusted block that might be
used something like this:
---
void foo(Range)(Range r) @trusted {
// ...
untrusted {
r.front;
}
// Your manually checked code.
untrusted {
r.popFront;
}
// …
}
---
This seems incredibly backwards. Not only is it confusing to
read, it also encourages bugs where operations mistakenly end up
being @trusted by forgetting to mark, say, an added call to
r.empty as untrusted. On the other hand, it is much more unlikely
that you accidentally add a new @trusted block. It seems obvious
that explicitly whitelisting a small number of potentially
dangerous but safe operations is much less error-prone approach
than disabling compiler checks for everything and then having to
remember to blacklist all unverified external dependencies.
David
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 16:19:26 UTC, John Colvin wrote:
On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu
wrote:
On 2/6/15 3:57 AM, Martin Krejcirik wrote:
If I understand it correctly, Walter is against adding
trusted blocks
(trusted {...}) into @safe functions. But what about having
safe blocks
in @trusted functions ?
That would be sensible - perhaps the best step forward
following this long discussion. -- Andrei
It feels inelegant, but it might be the best way out of a bad
situation.
I can instantly see this happening:
void foo() @trusted
{
@safe
{
//loads of code
}
//a few lines of system code, only safe due to context in
the @safe blocks
@safe
{
\\loads of code
}
}
Is that what we want? I can't see why not, but it feels off
somehow... Effectively you've got @trusted blocks in an
@trusted function, just inverted.
It's been suggested that '@system' be used to mark the blocks in
question. The point would be to emphasize the dangerousness of
the operation. The function is still @trusted, but inside, the
@system code is marked as such.
Re: @trust is an encapsulation method, not an escape
On 2/6/15 9:07 AM, Tobias Pankrath wrote: At least I expect the amount of @trusted in phobos to be much higher than in normal user code. Exactly. There's a bunch of low-level interfaces in Phobos and also some code that aims at maximum efficiency. Andrei
Re: @trust is an encapsulation method, not an escape
On 2/6/15 8:40 AM, David Nadlinger wrote:
On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu wrote:
On 2/6/15 3:57 AM, Martin Krejcirik wrote:
If I understand it correctly, Walter is against adding trusted blocks
(trusted {...}) into @safe functions. But what about having safe blocks
in @trusted functions ?
That would be sensible - perhaps the best step forward following this
long discussion. -- Andrei
This still does not solve the template inference problem
What is that?
though, unless
you make it a non-@trusted block instead of requiring @safe-ty. And
then you'd end up with the—at least to my eyes—rather absurd situation
that a template function that is marked @trusted might actually end up
being @system.
@trusted functions are @system.
Andrei
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 17:12:40 UTC, David Nadlinger wrote: Let's say you have a template function that accepts a range. For performance, you want to do some of the processing in a way that is @system, but can be verified to be correct for all inputs in this specific case. In other words, that piece of code can be rightfully @trusted. However, marking the whole function as @trusted would be a mistake, as the primitives of the range that is your input data might be @system. Using @trusted blocks (which is what is currently emulated by the nested functions/lambdas), you can just mark your unsafe code as @trusted and let the compiler figure out the safety of the whole function. @safe blocks wouldn't work for this, as you'd inadvertently require the user-supplied input range to have @safe/@trusted primitives. I'm trying to promote suggesting '@system' blocks instead of '@trusted'. '@trusted' functions, but '@system' blocks - which can only go in @trusted functions (@system block in @system functions are redundant). It's the same semantics, but it might win the day because the intent is to isolate the @system code, while still presenting a @trusted interface, as seems so important to the leadership.
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 10:28:38 UTC, Walter Bright wrote: On 2/6/2015 1:01 AM, Vladimir Panteleev wrote: On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote: Cue my endless attempts to explain the difference between input errors and logic errors :-( So which one is it? http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html That doesn't answer my question. A few years ago, I recall, you were arguing that for functions which are or may be exported to a DLL, thus all Phobos functions, it is impossible to predict how the functions will be used. Thus, you argued, the functions' input has to be validated, even if invalid parameters can only be passed to the function as a result of a program bug, and never user input. So, to repeat my question: which one is it? Have you changed your mind, or are there exceptions to the rules in the post you quoted?
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:
On 2/6/2015 12:31 AM, Kagamin wrote:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright
wrote:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Should be enforce: assert doesn't guard against malicious
usage.
Cue my endless attempts to explain the difference between input
errors and logic errors :-(
A little offtop: if this function is compiled in release mode and
compiler assumes assert holds, it's free to use dest.length
instead of src.length and if at runtime dest is longer than src,
this will create heartbleed-like bug in safe code.
Re: @trust is an encapsulation method, not an escape
First, I want to say that I didn't want to cause a huge rift between D
developers with this, I didn't think this was such a drastic issue, just
a possible idea. But here we are. I hope we can mend this, and move
forward. But on to the discussion.
On 2/5/15 6:39 PM, Walter Bright wrote:
Consider the following code excerpted from std.array.join:
static U trustedCast(U, V)(V v) @trusted { return cast(U) v; }
return trustedCast!RetType(result);
This is because the compiler would complain that the following line
would not be @safe:
return cast(RetType)(result);
The rationale is I know it's safe, so I'll create an @trusted wrapper
to eliminate the error. What comes next is that's cumbersome. How
about a better syntax:
trusted {
return cast(RetType)(result);
}
No. This was NEVER THE ARGUMENT.
Now, let me explain why the latter is BETTER.
It's better because I know where it is used. It's used in one place, and
I can squash it right there saying No, you can't do this in this one
place. Instead of reviewing an API in ALL POSSBILE CONTEXTS (which if
trustedCast is a public API, would be a lot), I have to review one call
in ONE CONTEXT.
The former is WORSE because it can be used in 100 places. Now I have to
go through and fix ALL THOSE FUNCTIONS that use it, because its
interface was exposed to the whole of phobos.
The problem, as we have said many times, is maintenance. Sure, in both
cases they never should have shown up in the first place. But there they
are, and we now have to fix them. All of them.
And let's also talk about long term maintenance. Any time a @trusted
function is amended or tweaked, we have to reconsider all the contexts.
If you mark a single line as @trusted, then additional lines to the same
function would not need as much scrutiny, especially if you warn when
@trusted tainted data is touched.
The trouble with it is, what if the cast is converting from an integer
to a pointer? That could lead to memory corruption. The code allows a
potentially memory corrupting operation to be inserted into code that is
otherwise @safe.
And so, you reject that code in both cases, except in the case where you
don't define a callable API, you don't have to worry about any other
code or contexts. This has to be the worst example to explain your point.
The only way to deal with it is to then manually review everything about
'RetType' and 'result' to prove to oneself that it is not converting
random bit patterns into pointers. In other words, one is manually
reviewing @safe code for memory corruption errors.
@trusted code is not @safe code. You are reviewing that one line in its
current context, not the whole function to see where it is called in
various contexts.
The solution is to regard @trusted as a means of encapsulating unsafe
operations, not escaping them. Encapsulating them means that the
interface from the @trusted code is such that it is usable from safe
code without having to manually review the safe code for memory safety.
For example (also from std.array):
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
I don't have to review callers of trustedMemory() because it
encapsulates an unsafe operation (memcpy) with a safe interface.
Sure. But let's consider it's called in one place:
trustedMemcopy(array[pos..pos+to_insert], tmp);
What if that became:
assert(tmp.length == to_insert); // same as src.length == dest.length
@trusted memcpy(array.ptr + pos, tmp.ptr, tmp.length);
What is missing here is, make sure the type of array and tmp is the
same. All one has to do is review the function to see that. You could
put in a static assert if you wish:
static assert(is(typeof(array) == typeof(tmp)));
If there are multiple places that it's called from, sure we can
encapsulate that in a function:
static void trustedMemcopy(T[] dest, T[] src)
{
assert(src.length == dest.length);
@trusted memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
There is very little difference here. Except if I add code to my version
of trustedMemcopy that is @system, I have to properly mark that too, and
that should imply greater scrutiny. I fail to see why making extra
unintended @system calls inside a trusted function shouldn't require
extra marking. Consider if this is a github review, and the context for
the new lines is missing (i.e. you don't see that the new line is inside
a @trusted function because github doesn't give you that line).
The reason @trusted applies only to functions, and not to blocks of
code, is that functions are the construct in D that provides an
interface. Arbitrary blocks of code do not have a structured interface.
Adding @trusted { code } support will encourage incorrect uses like the
opening example. The existence of @trusted blocks will require review of
every line of code in the function that
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 13:28:59 UTC, Steven Schveighoffer wrote: The bottom line of my reasoning is that code changes over time, by different people. Context is forgotten. It's much better to have the compiler verify you know what you are doing when working with @trusted than it is to just allow anyone to inject code anywhere. Actually, I think this argument goes against what you are arguing for. Anything within a @trusted section has a big warning sign attached to it that says cannot modify this without detailed review. But the compiler cannot verify that a @safe function with local @trusted blocks are actually safe, so it only buys you a false sense of security. It is also much easier to bring a large @trusted block to safety than a small one, e.g. to have one big @trusted chunk that does: 1. obtain resource 2. process resource 3. copy resource 4. free resource 5. return copy The problem is the process around @trusted given that there will be no sound verification system in D. Maybe it should have been called @manually_proven_safe instead, to discourage use...
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:
On 2/6/2015 12:31 AM, Kagamin wrote:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright
wrote:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
assert(src.length == dest.length);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Should be enforce: assert doesn't guard against malicious
usage.
Cue my endless attempts to explain the difference between input
errors and logic errors :-(
Well, ok, not enforce:
static void trustedMemcopy(T[] dest, T[] src) @trusted
{
if(src.length != dest.length)assert(false);
memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Re: @trust is an encapsulation method, not an escape
6. @trusted code should be self contained so that changes in called functions don't break the proof...
Re: @trust is an encapsulation method, not an escape
On 2/6/15 8:42 AM, Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= ola.fosheim.grostad+dl...@gmail.com wrote: On Friday, 6 February 2015 at 13:28:59 UTC, Steven Schveighoffer wrote: The bottom line of my reasoning is that code changes over time, by different people. Context is forgotten. It's much better to have the compiler verify you know what you are doing when working with @trusted than it is to just allow anyone to inject code anywhere. Actually, I think this argument goes against what you are arguing for. Anything within a @trusted section has a big warning sign attached to it that says cannot modify this without detailed review. But the compiler cannot verify that a @safe function with local @trusted blocks are actually safe, so it only buys you a false sense of security. That is kind of the point behind H.S. Teoh's idea that @trusted code should be default @safe with @system escapes instead of today where it's default @system. If nothing else, it has discouraged the use of @trusted to mark code for extra scrutiny. I'm coming to agree with him. Having a function marked @safe doesn't tell you whether there are @trusted blocks, and any @trusted blocks (despite the idea of having the compiler tell you when data was generated/touched by @trusted code) can bring into suspect the whole function. So marking a function @safe, and having it mean this function has NO TRUSTED OR SYSTEM CODE in it whatsoever, is probably the right move, regardless of any other changes. In fact, I would propose that it should be an error for @safe functions to have any non-static functions inside them (i.e. any place where the aliasing of @safe data is implicit). This is a departure from my previous position, but I think it still fits if we allow @trusted to mean what H.S. Teoh's proposal means. It is also much easier to bring a large @trusted block to safety than a small one, e.g. to have one big @trusted chunk that does: 1. obtain resource 2. process resource 3. copy resource 4. free resource 5. return copy The problem is the process around @trusted given that there will be no sound verification system in D. What we really want is: 1. A way to say this function needs extra scrutiny 2. Mechanical verification as MUCH AS POSSIBLE, and especially for changes to said function. Yes, we can do 2 manually if necessary. But having a compiler that never misses on pointing out certain bad things is so much better than not having it. Maybe it should have been called @manually_proven_safe instead, to discourage use... @assume_safe would probably be the right moniker since that's what we use elsewhere. But it's water under the bridge now... -Steve
Re: @trust is an encapsulation method, not an escape
On Friday, 6 February 2015 at 13:42:40 UTC, Ola Fosheim Grøstad wrote: cannot modify this without detailed review. This quote from Ola, here? That basically describes my job maintaining big piles of legacy C: the compiler verifies nothing, so every change to the anything in the API of safe functions or anything in their entire call chain must be painstakingly reviewed. A single change generally takes me several days of research, verification, and testing. I fixed about 150 potential memory issues (and several really dumb logic errors) with Clang's static analysis when I first inherited the code; it took weeks. (And now writing new stuff using this safe API is turning up memory safety issues anyway!) So from my perspective, calling this situation completely impractical reveals a stunning gift for understatement. Is this really the best we can do after however many years? Because it blows. The current @trusted semantics (and accompanying politics) make it exceedingly clear that @safe is meaningless for anything beyond trivial, one-off tools that will never receive maintenance. -Wyatt
