Re: shared - i need it to be useful
On Mon, Oct 22, 2018 at 6:00 AM Timon Gehr via Digitalmars-d
wrote:
>
> On 22.10.18 12:26, Timon Gehr wrote:
> > ---
> > module borked;
> >
> > void atomicIncrement(int* p)@system{
> > import core.atomic;
> > atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
> > }
> >
> > struct Atomic(T){
> > private T val;
> > void opUnary(string op : "++")() shared @trusted {
> > atomicIncrement(cast(T*));
> > }
> > }
> > void main()@safe{
> > Atomic!int i;
> > auto a=&[i][0];// was: Atomic!int* a =
> > import std.concurrency;
> > spawn((shared(Atomic!int)* a){ ++*a; }, a);
> > ++i.val; // race
> > }
> > ---
>
> Obviously, this should have been:
>
> ---
> module borked;
>
> void atomicIncrement(int*p)@system{
> import core.atomic;
> atomicOp!"+="(*cast(shared(int)*)p,1);
> }
> struct Atomic(T){
> private T val;
> void opUnary(string op:"++")()shared @trusted{
> atomicIncrement(cast(T*));
> }
> }
> void main()@safe{
> auto a=new Atomic!int;
> import std.concurrency;
> spawn((shared(Atomic!int)* a){ ++*a; }, a);
> ++a.val; // race
> }
> ---
>
> (I was short on time and had to fix Manu's code because it was not
> actually compilable.)
Nitpick; atomicOp does not receive a shared arg under my proposal,
it's not a threadsafe function by definition as discussed a few times.
Re: shared - i need it to be useful
On Mon, Oct 22, 2018 at 4:50 AM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
>
> > No no, they're repeated, not scattered, because I seem to have
> > to keep repeating it over and over, because nobody is reading
> > the text, or perhaps imaging there is a lot more text than
> > there is.
> > ...
> > You mean like every post in opposition which disregards the
> > rules and baselessly asserts it's a terrible idea? :/
> > ...
> > I responded to your faulty program directly with the correct
> > program, and you haven't acknowledged it. Did you see it?
>
> Right back at you.
>
> Quote:
>
> I think this is a typical sort of construction:
>
> struct ThreadsafeQueue(T)
> {
>void QueueItem(T*) shared;
>T* UnqueueItem() shared;
> }
>
> struct SpecialWorkList
> {
>struct Job { ... }
>
>void MakeJob(int x, float y, string z) shared // <- any thread
> may
> produce a job
>{
> Job* job = new Job; // <- this is thread-local
> PopulateJob(job, x, y, z); // <- preparation of a job might be
> complex, and worthy of the SpecialWorkList implementation
>
> jobList.QueueItem(job); // <- QueueItem encapsulates
> thread-safety, no need for blunt casts
>}
>
>void Flush() // <- not shared, thread-local consumer
>{
> Job* job;
> while (job = jobList.UnqueueItem()) // <- it's obviously safe
> for
> a thread-local to call UnqueueItem even though the implementation
> is
> threadsafe
> {
>// thread-local dispatch of work...
>// perhaps rendering, perhaps deferred destruction, perhaps
> deferred resource creation... whatever!
> }
>}
>
>void GetSpecialSystemState() // <- this has NOTHING to do with
> the
> threadsafe part of SpecialWorkList
>{
> return os.functionThatChecksSystemState();
>}
>
>// there may be any number of utility functions that don't
> interact
> with jobList.
>
> private:
>void PopulateJob(ref Job job, ...)
>{
> // expensive function; not thread-safe, and doesn't have any
> interaction with threading.
>}
>
>ThreadsafeQueue!Job jobList;
> }
>
>
> This isn't an amazing example, but it's typical of a thing that's
> mostly thread-local, and only a small controlled part of it's
> functionality is thread-safe.
> The thread-local method Flush() also deals with thread-safety
> internally... because it flushes a thread-safe queue.
>
> All thread-safety concerns are composed by a utility object, so
> there's no need for locks, magic, or casts here.
>
> EndQuote;
>
> The above:
> 1) Will not compile, not currently, not under your proposal
> (presumably you forgot in frustration to cast before calling
> PopulateJob?..)
I did correct that line (along with an apology) on my very next post;
it would probably be a member of Job... or any manner of other code.
That is the least interesting line in the program
> 2) Does not in any way demonstrate a practical @safe application
> of an implicit conversion. As I wrote in the original response to
> that code, with that particular code it seems more like you just
> need forwarding methods that call `shared` methods under the hood
> (i.e. MakeJob), and it'd be "nice" if you didn't have to write
> those and could just call `shared` MakeJob on an un-`shared`
> reference directly. But these are all assumptions without seeing
> the actual usage.
>
> Please just stop acting like everyone here is opposing *you*.
You're right, it's mostly you.
> All
> you're doing is dismissing everyone with a "nuh-huh, you no
> understand, you bad". If it was just me, fine, it would mean I'm
> dumb and not worthy of this discussion. But this isn't the case,
> which means *you are not getting your point across*. And yet
> instead of trying to fix that, you're getting all snarky.
I mean, it's fair, but it's pretty bloody hypocritical coming from you!
I think it's fair to point out that your high-frequency, persistent,
and unwavering hostility from post #1 across all my recent threads (at
least, until I told you to GF) is the primary reason I'm frustrated
here.
You can own part responsibility for my emotion.
Re: shared - i need it to be useful
On Mon, Oct 22, 2018 at 3:30 AM Timon Gehr via Digitalmars-d
wrote:
>
> On 22.10.18 02:54, Manu wrote:
> > On Sun, Oct 21, 2018 at 5:40 PM Timon Gehr via Digitalmars-d
> > wrote:
> >>
> >> On 21.10.18 21:04, Manu wrote:
> >>> On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
> >>> wrote:
>
> On 21.10.18 17:54, Nicholas Wilson wrote:
> >
> >> As soon as that is done, you've got a data race with the other
> >> existing unshared aliases.
> >
> > You're in @trusted code, that is the whole point. The onus is on the
> > programmer to make that correct, same with regular @safe/@trusted@system
> > code.
>
> Not all of the parties that participate in the data race are in @trusted
> code. The point of @trusted is modularity: you manually check @trusted
> code according to some set of restrictions and then you are sure that
> there is no memory corruption.
>
> Note that you are not allowed to look at any of the @safe code while
> checking your @trusted code. You will only see an opaque interface to
> the @safe code that you call and all you know is that all the @safe code
> type checks according to @safe rules. Note that there might be an
> arbitrary number of @safe functions and methods that you do not see.
>
> Think about it this way: you first write all the @trusted and @system
> code, and some evil guy who does not like you comes in after you looks
> at your code and writes all the @safe code. If there is any memory
> corruption, it will be your fault and you will face harsh consequences.
> Now, design the @safe type checking rules. It won't be MP!
>
> Note that there may well be a good way to get the good properties of MP
> without breaking the type system, but MP itself is not good because it
> breaks @safe.
> >>>
> >>> Show me. Nobody has been able to show that yet. I'd really like to know
> >>> this.
> >>>
> >>
> >> I just did,
> >
> > There's no code there... just a presumption that the person who wrote
> > the @trusted code did not deliver the promise they made.
> > ...
>
> Yes, because there is no way to write @trusted code that holds its
> promise while actually doing something interesting in multiple threads
> if @safe code can implicitly convert from unshared to shared.
How do my examples prior fail to hold their promise?
struct S
{
private int x;
void method() shared @trusted { /* safely manipulate x */ }
}
How can you not trust that function? How can a 3rd party invalidate
that functions promise? `x` is inaccessible.
S can be thread-local or shared as much as you like, and it's safe,
and I don't know how a 3rd party could @safely undermine that?
> >> but if you really need to, give me a non-trivial piece of> correct
> >> multithreaded code that accesses some declared-unshared field
> >> from a shared method and I will show you how the evil guy would modify
> >> some @safe code in it and introduce race conditions. It needs to be your
> >> code, as otherwise you will just claim again that it is me who wrote bad
> >> @trusted code.
> >
> > You can pick on any of my prior code fragments. They've all been ignored so
> > far.
> >
>
> I don't want "code fragments". Show me the real code.
>
> I manually browsed through posts now (thanks a lot) and found this
> implementation:
>
> struct Atomic(T){
>void opUnary(string op : "++")() shared { atomicIncrement(); }
>private T val;
> }
>
> This is @system code. There is no @safe or @trusted here, so I am
> ignoring it.
>
>
> Then I browsed some more, because I had nothing better to do, and I
> found this. I completed it so that it is actually compilable, except for
> the unsafe implicit conversion.
>
> Please read this code, and then carefully read the comments below it
> before you respond. I will totally ignore any of your answers that
> arrive in the next two hours.
>
> ---
> module borked;
>
> void atomicIncrement(int* p)@system{
> import core.atomic;
> atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
> }
>
> struct Atomic(T){
> private T val;
> void opUnary(string op : "++")() shared @trusted {
> atomicIncrement(cast(T*));
> }
> }
> void main()@safe{
> Atomic!int i;
> auto a=&[i][0];// was: Atomic!int* a =
> import std.concurrency;
> spawn((shared(Atomic!int)* a){ ++*a; }, a);
> ++i.val; // race
> }
> ---
>
>
> Oh no! The author of the @trusted function (i.e. you) did not deliver on
> the promise they made!
>
> Now, before you go and tell me that I am stupid because I wrote bad
> code, consider the following:
>
> - It is perfectly @safe to access private members from the same module.
>
> - You may not blame the my @safe main function for the problem. It is
> @safe, so it cannot be blamed for UB. Any UB is the result of a bad
> @trusted function, a compiler bug, or hardware failure.
>
> - The only @trusted function in this
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 14:31:28 UTC, Timon Gehr wrote:
On 22.10.18 16:09, Simen Kjærås wrote:
On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
module reborked;
import atomic;
void main()@safe{
auto a=new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++a.tupleof[0];
}
Finally! Proof that MP is impossible. On the other hand, why
the hell is that @safe? It breaks all sorts of guarantees
about @safety. At a minimum, that should be un-@safe.
Filed in bugzilla:
https://issues.dlang.org/show_bug.cgi?id=19326
--
Simen
Even if this is changed (and it probably should be), it does
not fix the case where the @safe function is in the same
module. I don't think it is desirable to change the definition
of @trusted such that you need to check the entire module if it
contains a single @trusted function.
If I can break safety of some (previously correct) code by
editing only @safe code, then that's a significant blow to
@safe. I think we need a general way to protect data from being
manipulated in @safe code in any way, same module or not.
What do you mean by 'previously correct'?
struct Array(T) {
@safe:
private int* ptr;
private int length;
@disable this();
this(int n) @trusted {
ptr = new int[n].ptr;
length = n;
foreach (ref e; ptr[0..length])
e = 123;
}
@trusted ref int get(int idx) {
assert(idx < length);
return ptr[idx];
}
}
unittest {
auto s = Array!int(1);
assert(s.get(0) == 123);
}
Is this correct code?
What if I add this:
@safe void bork(T)(ref Array!T s) {
s.length *= 2;
}
unittest {
auto s = Array!int(1);
bork(s);
assert(s.get(1) == 123); // Out of bounds!
}
--
Simen
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d
wrote:
On 10/21/2018 2:08 PM, Walter Bright wrote:
> On 10/21/2018 12:20 PM, Nicholas Wilson wrote:
>> Yes, but the problem you describe is arises from implicit
>> conversion in the other direction, which is not part of the
>> proposal.
>
> It's Manu's example.
Then I don't know what the proposal is. Pieces of it appear to
be scattered over numerous posts, mixed in with other text,
No no, they're repeated, not scattered, because I seem to have
to keep repeating it over and over, because nobody is reading
the text, or perhaps imaging there is a lot more text than
there is.
I can go look at the original post - and I have - but while it
may strictly speaking contain all the information I need, the
amount of time and brain power it would take for me to comprehend
the consequences is pretty large. I assume you have done a lot of
that work already and could save everyone a lot of time by
putting together a quick document that covers some of that, with
good examples. I'm not going to read {1,2,3}00 messages full of
irritated bidirectional miscommunication to try and understand
this unless I really have to, and I assume others feel similarly.
Re: shared - i need it to be useful
On 22.10.18 16:09, Simen Kjærås wrote:
On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
module reborked;
import atomic;
void main()@safe{
auto a=new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++a.tupleof[0];
}
Finally! Proof that MP is impossible. On the other hand, why the hell is
that @safe? It breaks all sorts of guarantees about @safety. At a
minimum, that should be un-@safe.
Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326
--
Simen
Even if this is changed (and it probably should be), it does not fix the
case where the @safe function is in the same module. I don't think it is
desirable to change the definition of @trusted such that you need to
check the entire module if it contains a single @trusted function.
If I can break safety of some (previously correct) code by editing only
@safe code, then that's a significant blow to @safe. I think we need a
general way to protect data from being manipulated in @safe code in any
way, same module or not.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote: On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d wrote: Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text, No no, they're repeated, not scattered, because I seem to have to keep repeating it over and over, because nobody is reading the text, or perhaps imaging there is a lot more text than there is. I've read every post on this thread and I also have the feeling that it's scattered. At the very least, I'm 90% confident I don't understand what it is you're proposing. Trust me, I'm trying. I believe that you have a proposal which you believe results in @safe multithreaded code. I don't understand how what I've read so far would accomplish that. I'm conviced that shared data shouldn't be allowed to be written to, but I haven't yet been convinced of anything else. I don't see how it's possible that implicit conversion from non-shared to shared can work at all. Yes, I know that in the proposal putting `shared` on anything makes it useless, but *somehow* that data gets to be used, even if it's by a @trusted function that casts away shared. At that point, nothing you do thread-safely to the shared data matters if you obtained the shared data from an implicit conversion. There may be many many aliases to it before it was converted, all of them able to write to that memory location believing it's not shared. And it would be @safe (but definitely not thread-safe) to do so! This has been explained a few times, by multiple people. I haven't seen anyone addressing this yet (it's possible it got lost in a sea of text). I don't even understand why it is you want to cast anything to shared anyway - that'd always be a code smell if I saw it during code review. If it's shared, type it as such. Or better yet, if you can just use immutable. I understand the frustration of not getting your point across. I would like to kindly point out that, if the replies have gotten to multiple dozen pages and several well-meaning people still don't get it, then the proposal probably isn't as simple as you believe it to be.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 13:40:39 UTC, Timon Gehr wrote:
module reborked;
import atomic;
void main()@safe{
auto a=new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++a.tupleof[0];
}
Finally! Proof that MP is impossible. On the other hand, why the
hell is that @safe? It breaks all sorts of guarantees about
@safety. At a minimum, that should be un-@safe.
Filed in bugzilla: https://issues.dlang.org/show_bug.cgi?id=19326
--
Simen
Re: shared - i need it to be useful
On 22.10.18 15:26, Simen Kjærås wrote:
Here's the correct version:
module atomic;
void atomicIncrement(int* p) @system {
import core.atomic;
atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}
struct Atomic(T) {
// Should probably mark this shared for extra safety,
// but it's not strictly necessary
private T val;
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*));
}
}
-
module unborked;
import atomic;
void main() @safe {
auto a = new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
//++i.val; // Cannot access private member
}
Once more, Joe Average Programmer should not be writing the @trusted
code in Atomic!T.opUnary - he should be using libraries written by
people who have studied the exact issues that make multithreading hard.
--
Simen
module reborked;
import atomic;
void main()@safe{
auto a=new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++a.tupleof[0];
}
Re: shared - i need it to be useful
On 22.10.18 03:01, Manu wrote: On Sun, Oct 21, 2018 at 5:55 PM Timon Gehr via Digitalmars-d wrote: On 22.10.18 02:45, Manu wrote: On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d wrote: On 21.10.18 20:46, Manu wrote: Shared data is only useful if, at some point, it is read/written, presumably by casting it to unshared in @trusted code. As soon as that is done, you've got a data race with the other existing unshared aliases. If such a race is possible, then the @trusted function is not threadsafe, so it is not @trusted by definition. You wrote a bad @trusted function, and you should feel bad. ... I wonder where this "each piece of code is maintained by only one person and furthermore this is the only person that will suffer if the code has bugs" mentality comes from. It is very popular as well as obviously nonsense. The simplest way to guarantee that no unsafe access is possible is to use encapsulation to assure no unregulated access exists. This only works if untrusted programmers (i.e. programmers who are only allowed to write/modify @safe code) are not allowed to change your class. I.e. it does not work. Have you ever cracked open std::map and 'fixed' it because you thought it was bad? (Also, yes, some people do that because std::map does not provide an interface to augment the binary search tree.) Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic without understanding what they're doing. You are not proposing to let core.atomic.Atomic convert to shared implicitly, you are proposing to do that for all classes. You can always implicitly convert to shared. Yes, exactly what I said. Where did I ever say anything like that? I'm sure I've never said this. ??? I said that you are proposing to allow implicit conversions to shared for all classes, not only core.atomic.Atomic, and the last time you said it was the previous sentence of the same post. How do these transformations of what I've said keep happening? ... You literally said that nobody changes core.atomic.Atomic. Anyway, even if I bought that @safe somehow should not be checked within druntime (I don't), bringing up this example does not make for a coherent argument why implicit conversion to shared should be allowed for all classes. You seem to be stuck on the detail whether you can trust the @trusted author though... Again: the @safe author is the problem. I don't follow. The @safe author is incapable of doing threadsafety violation. They are capable of doing so as soon as you provide them a @trusted function that treats data as shared that they can access as unshared. They can only combine threadsafe functions. They can certainly produce a program that doesn't work, and they are capable of ordering issues, but that's not the same as data-race related crash bugs. Accessing private members of aggregates in the same module is @safe. tupleof is @safe too.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 10:26:14 UTC, Timon Gehr wrote:
module borked;
void atomicIncrement(int* p)@system{
import core.atomic;
atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}
struct Atomic(T){
private T val;
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*));
}
}
void main()@safe{
Atomic!int i;
auto a=&[i][0];// was: Atomic!int* a =
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++i.val; // race
}
---
Oh no! The author of the @trusted function (i.e. you) did not
deliver on the promise they made!
Now, before you go and tell me that I am stupid because I wrote
bad code, consider the following:
- It is perfectly @safe to access private members from the same
module.
Yes, so you need to place the @trusted code in a separate module.
The @trusted code will be very rare (Atomic!T, lockfree queue,
mutex, semaphore...). This will be in the standard library,
possibly some other library. You should not be writing your own
implementations of these. You should not be writing @trusted code
without very good reason. You should be using @safe functions all
over your code if at all possible.
- You may not blame the my @safe main function for the problem.
It is @safe, so it cannot be blamed for UB. Any UB is the
result of a bad @trusted function, a compiler bug, or hardware
failure.
No, we blame the fact you have not blocked off non-thread-safe
access to Atomic!T.val. What you have made is this:
https://i.imgur.com/PnKMigl.jpg
The piece of code to blame is the @trusted function - you can't
trust it, since non-thread-safe access to Atomic!T.val has not
been blocked off. As long as anyone can extend its interface,
Atomic!T can't be thread-safe.
I'll quote myself:
For clarity: the interface of a type is any method, function,
delegate or otherwise that may affect its internals. That
means any free function in the same module, and any
non-private members.
I've actually missed some possibilities there - member functions
of other types in the same module must also count as part of the
interface. Because of this wide net, modules that implement
thread-safe types with shared methods should be short and sweet.
- The only @trusted function in this module was written by you.
You said that there is a third implementation somewhere. If
that one actually works, I apologize and ask you to please
paste it again in this subthread.
Here's the correct version:
module atomic;
void atomicIncrement(int* p) @system {
import core.atomic;
atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}
struct Atomic(T) {
// Should probably mark this shared for extra safety,
// but it's not strictly necessary
private T val;
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*));
}
}
-
module unborked;
import atomic;
void main() @safe {
auto a = new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
//++i.val; // Cannot access private member
}
Once more, Joe Average Programmer should not be writing the
@trusted code in Atomic!T.opUnary - he should be using libraries
written by people who have studied the exact issues that make
multithreading hard.
--
Simen
Re: shared - i need it to be useful
On 22.10.18 14:39, Aliak wrote:
On Monday, 22 October 2018 at 10:26:14 UTC, Timon Gehr wrote:
---
module borked;
void atomicIncrement(int* p)@system{
import core.atomic;
atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}
struct Atomic(T){
private T val;
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*));
}
}
void main()@safe{
auto a=new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++a.val; // race
}
---
Oh no! The author of the @trusted function (i.e. you) did not deliver
on the promise they made!
hi, if you change the private val in Atomic to be “private shared T
val”, is the situation the same?
It's a bit different, because then there is no implicit unshared->shared
conversion happening, and this discussion is only about that. However,
without further restrictions, you can probably construct cases where a
@safe function in one module escapes a private shared(T)* member to
somewhere else that expects a different synchronization strategy.
Therefore, even if we agree that unshared->shared conversion cannot be
implicit in @safe code, the 'shared' design is not complete, but it
would be a good first step to agree that this cannot happen, such that
we can then move on to harder issues.
E.g. probably it would be good to have something like @trusted data that
cannot be manipulated from @safe code, such that @trusted functions can
rely on some invariants.
Re: shared - i need it to be useful
On 22.10.18 12:26, Timon Gehr wrote:
---
module borked;
void atomicIncrement(int* p)@system{
import core.atomic;
atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}
struct Atomic(T){
private T val;
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*));
}
}
void main()@safe{
Atomic!int i;
auto a=&[i][0];// was: Atomic!int* a =
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++i.val; // race
}
---
Obviously, this should have been:
---
module borked;
void atomicIncrement(int*p)@system{
import core.atomic;
atomicOp!"+="(*cast(shared(int)*)p,1);
}
struct Atomic(T){
private T val;
void opUnary(string op:"++")()shared @trusted{
atomicIncrement(cast(T*));
}
}
void main()@safe{
auto a=new Atomic!int;
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++a.val; // race
}
---
(I was short on time and had to fix Manu's code because it was not
actually compilable.)
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 10:26:14 UTC, Timon Gehr wrote:
---
module borked;
void atomicIncrement(int* p)@system{
import core.atomic;
atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}
struct Atomic(T){
private T val;
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*));
}
}
void main()@safe{
Atomic!int i;
auto a=&[i][0];// was: Atomic!int* a =
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++i.val; // race
}
---
Oh no! The author of the @trusted function (i.e. you) did not
deliver on the promise they made!
hi, if you change the private val in Atomic to be “private shared
T val”, is the situation the same?
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote:
No no, they're repeated, not scattered, because I seem to have
to keep repeating it over and over, because nobody is reading
the text, or perhaps imaging there is a lot more text than
there is.
...
You mean like every post in opposition which disregards the
rules and baselessly asserts it's a terrible idea? :/
...
I responded to your faulty program directly with the correct
program, and you haven't acknowledged it. Did you see it?
Right back at you.
Quote:
I think this is a typical sort of construction:
struct ThreadsafeQueue(T)
{
void QueueItem(T*) shared;
T* UnqueueItem() shared;
}
struct SpecialWorkList
{
struct Job { ... }
void MakeJob(int x, float y, string z) shared // <- any thread
may
produce a job
{
Job* job = new Job; // <- this is thread-local
PopulateJob(job, x, y, z); // <- preparation of a job might be
complex, and worthy of the SpecialWorkList implementation
jobList.QueueItem(job); // <- QueueItem encapsulates
thread-safety, no need for blunt casts
}
void Flush() // <- not shared, thread-local consumer
{
Job* job;
while (job = jobList.UnqueueItem()) // <- it's obviously safe
for
a thread-local to call UnqueueItem even though the implementation
is
threadsafe
{
// thread-local dispatch of work...
// perhaps rendering, perhaps deferred destruction, perhaps
deferred resource creation... whatever!
}
}
void GetSpecialSystemState() // <- this has NOTHING to do with
the
threadsafe part of SpecialWorkList
{
return os.functionThatChecksSystemState();
}
// there may be any number of utility functions that don't
interact
with jobList.
private:
void PopulateJob(ref Job job, ...)
{
// expensive function; not thread-safe, and doesn't have any
interaction with threading.
}
ThreadsafeQueue!Job jobList;
}
This isn't an amazing example, but it's typical of a thing that's
mostly thread-local, and only a small controlled part of it's
functionality is thread-safe.
The thread-local method Flush() also deals with thread-safety
internally... because it flushes a thread-safe queue.
All thread-safety concerns are composed by a utility object, so
there's no need for locks, magic, or casts here.
EndQuote;
The above:
1) Will not compile, not currently, not under your proposal
(presumably you forgot in frustration to cast before calling
PopulateJob?..)
2) Does not in any way demonstrate a practical @safe application
of an implicit conversion. As I wrote in the original response to
that code, with that particular code it seems more like you just
need forwarding methods that call `shared` methods under the hood
(i.e. MakeJob), and it'd be "nice" if you didn't have to write
those and could just call `shared` MakeJob on an un-`shared`
reference directly. But these are all assumptions without seeing
the actual usage.
Please just stop acting like everyone here is opposing *you*. All
you're doing is dismissing everyone with a "nuh-huh, you no
understand, you bad". If it was just me, fine, it would mean I'm
dumb and not worthy of this discussion. But this isn't the case,
which means *you are not getting your point across*. And yet
instead of trying to fix that, you're getting all snarky.
Re: shared - i need it to be useful
On 22.10.18 02:54, Manu wrote:
On Sun, Oct 21, 2018 at 5:40 PM Timon Gehr via Digitalmars-d
wrote:
On 21.10.18 21:04, Manu wrote:
On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d
wrote:
On 21.10.18 17:54, Nicholas Wilson wrote:
As soon as that is done, you've got a data race with the other
existing unshared aliases.
You're in @trusted code, that is the whole point. The onus is on the
programmer to make that correct, same with regular @safe/@trusted@system
code.
Not all of the parties that participate in the data race are in @trusted
code. The point of @trusted is modularity: you manually check @trusted
code according to some set of restrictions and then you are sure that
there is no memory corruption.
Note that you are not allowed to look at any of the @safe code while
checking your @trusted code. You will only see an opaque interface to
the @safe code that you call and all you know is that all the @safe code
type checks according to @safe rules. Note that there might be an
arbitrary number of @safe functions and methods that you do not see.
Think about it this way: you first write all the @trusted and @system
code, and some evil guy who does not like you comes in after you looks
at your code and writes all the @safe code. If there is any memory
corruption, it will be your fault and you will face harsh consequences.
Now, design the @safe type checking rules. It won't be MP!
Note that there may well be a good way to get the good properties of MP
without breaking the type system, but MP itself is not good because it
breaks @safe.
Show me. Nobody has been able to show that yet. I'd really like to know this.
I just did,
There's no code there... just a presumption that the person who wrote
the @trusted code did not deliver the promise they made.
...
Yes, because there is no way to write @trusted code that holds its
promise while actually doing something interesting in multiple threads
if @safe code can implicitly convert from unshared to shared.
but if you really need to, give me a non-trivial piece of> correct
multithreaded code that accesses some declared-unshared field
from a shared method and I will show you how the evil guy would modify
some @safe code in it and introduce race conditions. It needs to be your
code, as otherwise you will just claim again that it is me who wrote bad
@trusted code.
You can pick on any of my prior code fragments. They've all been ignored so far.
I don't want "code fragments". Show me the real code.
I manually browsed through posts now (thanks a lot) and found this
implementation:
struct Atomic(T){
void opUnary(string op : "++")() shared { atomicIncrement(); }
private T val;
}
This is @system code. There is no @safe or @trusted here, so I am
ignoring it.
Then I browsed some more, because I had nothing better to do, and I
found this. I completed it so that it is actually compilable, except for
the unsafe implicit conversion.
Please read this code, and then carefully read the comments below it
before you respond. I will totally ignore any of your answers that
arrive in the next two hours.
---
module borked;
void atomicIncrement(int* p)@system{
import core.atomic;
atomicOp!("+=",int,int)(*cast(shared(int)*)p,1);
}
struct Atomic(T){
private T val;
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*));
}
}
void main()@safe{
Atomic!int i;
auto a=&[i][0];// was: Atomic!int* a =
import std.concurrency;
spawn((shared(Atomic!int)* a){ ++*a; }, a);
++i.val; // race
}
---
Oh no! The author of the @trusted function (i.e. you) did not deliver on
the promise they made!
Now, before you go and tell me that I am stupid because I wrote bad
code, consider the following:
- It is perfectly @safe to access private members from the same module.
- You may not blame the my @safe main function for the problem. It is
@safe, so it cannot be blamed for UB. Any UB is the result of a bad
@trusted function, a compiler bug, or hardware failure.
- The only @trusted function in this module was written by you.
You said that there is a third implementation somewhere. If that one
actually works, I apologize and ask you to please paste it again in this
subthread.
Re: shared - i need it to be useful
On Mon, Oct 22, 2018 at 2:30 AM Walter Bright via Digitalmars-d wrote: > > On 10/22/2018 1:34 AM, Manu wrote: > > I posted it, twice... 2 messages, back to back, and you're responding > > to this one, and not that one. I'll post it again... > > > Posting it over and over is illustrative of the failure of posting proposal > documents to the n.g. instead of posting it as a DIP which can be referred to: > > 1. nobody knows which of your 70 messages are the ones with the proposal in it > > 2. with multiple posts of the proposal, nobody knows which one is the most > up-to-date one It hasn't changed. Not one single bit. I haven't changed a single detail in this thread. > Doing it this way does not work. Continuing to repost it is a waste of your > time. Post it as a DIP and link to it. And you STILL ignored my post >_< Please look at the proper code that implements your broken example. I know you've seen it now.
Re: shared - i need it to be useful
On 22/10/2018 10:28 PM, Walter Bright wrote: On 10/22/2018 1:34 AM, Manu wrote: I posted it, twice... 2 messages, back to back, and you're responding to this one, and not that one. I'll post it again... Posting it over and over is illustrative of the failure of posting proposal documents to the n.g. instead of posting it as a DIP which can be referred to: 1. nobody knows which of your 70 messages are the ones with the proposal in it 2. with multiple posts of the proposal, nobody knows which one is the most up-to-date one Doing it this way does not work. Continuing to repost it is a waste of your time. Post it as a DIP and link to it. As I've said previously, it doesn't need to be a good DIP or anywhere near complete. It just needs code examples comparing current and proposed behavior with some text about semantics changes.
Re: shared - i need it to be useful
On 10/22/2018 1:34 AM, Manu wrote: I posted it, twice... 2 messages, back to back, and you're responding to this one, and not that one. I'll post it again... Posting it over and over is illustrative of the failure of posting proposal documents to the n.g. instead of posting it as a DIP which can be referred to: 1. nobody knows which of your 70 messages are the ones with the proposal in it 2. with multiple posts of the proposal, nobody knows which one is the most up-to-date one Doing it this way does not work. Continuing to repost it is a waste of your time. Post it as a DIP and link to it.
Re: shared - i need it to be useful
On 10/22/2018 1:42 AM, Manu wrote: You removed whatever comment you're referring to. If your newsreader cannot find the antecedent, you badly need to use a better one. Thunderbird handles this rather well, there's no reason to use an inferior one. Or just click the <- button: https://digitalmars.com/d/archives/digitalmars/D/shared_-_i_need_it_to_be_useful_320165.html#N320607 I don't understand any of Timon's posts in this thread at all, which is unusual because he's usually pretty clear. I suspect Timon is equally frustrated at not getting his point across.
Re: shared - i need it to be useful
On Mon, Oct 22, 2018 at 12:55 AM Walter Bright via Digitalmars-d wrote: > > On 10/21/2018 11:58 AM, Timon Gehr wrote: > > [...] > > Thank you, Timon, for a nice explanation of what I was trying to express. You removed whatever comment you're referring to. I don't understand any of Timon's posts in this thread at all, which is unusual because he's usually pretty clear.
Re: shared - i need it to be useful
Third time's the charm maybe?
- repeated, 3rd time
On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d,
wrote:
>
> On 10/20/2018 11:24 AM, Manu wrote:
> > This is an unfair dismissal.
>
> It has nothing at all to do with fairness. It is about what the type system
> guarantees in @safe code. To repeat, the current type system guarantees in
> @safe
> code that T* and shared(T)* do not point to the same memory location.
>
> Does your proposal maintain that or not? It's a binary question.
By the definition Nick pulled from Wikipedia and posted for you a few
posts back, yes, my proposal satisfies Wikipedia's definition of no
aliasing. I understand that property is critical, and I have carefully
designed for it.
> > I'm not sure you've understood the proposal.
> > This is the reason for the implicit conversion. It provides safe
> > transition.
>
> I don't see any way to make an implicit T* to shared(T)* safe, or vice versa.
> The T* code can create more aliases that the conversion doesn't know about,
> and
> the shared(T)* code can hand out aliases to other threads. So it all falls to
> pieces.
T* can't make additional T* aliases on other threads; there can only
be one thread with T*.
shared(T)* can not make a T*.
shared(T)* has no read or write access, so it's not an alias of T* by
Wikipedia's definition.
Only threadsafe functions can do anything to T.
The leap of faith is; some @trusted utility functions at the bottom of
the shared stack makes a promise that it is threadsafe, and must
deliver that promise.
I don't think this is unreasonable; this is the nature of @trusted
functions, they make a promise, and they must keep it.
If the trusted function does not lie, then the chain of trust holds
upwards through the stack.
The are very few such trusted functions in practise. Like, similar to
the number of digits you have.
> Using a 'scope' qualifier won't work, because 'scope' isn't transitive,
> while shared is, i.e. U** and shared(U*)*.
I don't think I depend on scope in any way.
That was an earlier revision of thinking in an older thread.
> > I'm not sure how to clarify it, what can I give you?
>
> Write a piece of code that does such an implicit conversion that you argue is
> @safe. Make the code as small as possible. Your example:
>
> > int* a;
> > shared(int)* b = a;
>
> This is not safe.
>
> Manu's Proposal ---
> @safe:
> int i;
> int* a =
> StartNewThread(a); // Compiles! Coder has no idea!
>
> ... in the new thread ...
> void StartOfNewThread(shared(int)* b) {
>
> ... we have two threads accessing 'i',
> one thinks it is shared, the other unshared,
> and StartOfNewThread() has no idea and anyone
> writing code for StartOfNewThread() has no way
> to know anything is wrong ...
>
> lockedIncrement(b); // Data Race!
> }
This program doesn't compile. You receive an error because it is not safe.
The function is `lockedIncrement(int*)`. It can't receive a shared
argument; the function is not threadsafe by my definition.
You have written a program that produces the expected error that
alerts you that you have tried to do un-@safe and make a race.
Stanislav produced this same program, and I responded with the correct
program a few posts back.
I'll repeat it here; the @safe program to model this interaction is:
@safe:
// function is NOT threadsafe by my definition, can not be called on
shared arguments
void atomicIncrement(int*);
struct Atomic(T)
{
// encapsulare the unsafe data so it's inaccessible by any unsafe means
private T val;
// perform the unsafe cast in a trusted function
// we are able to assure a valid calling context by encapsulating
the data above
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*)); }
}
Atomic!int i;
Atomic!int* a =
StartNewThread(a); // Compiles, of course!
++i; // no race
... in the new thread ...
void StartOfNewThread(shared(Atomic!int)* b) {
//... we have two threads accessing 'i', one has thread-local
access, this one has a restricted shared access.
// here, we have a shared instance, so we can only access `b` via
threadsafe functions.
// as such, we can manipulate `b` without fear.
++i; // no race!
}
> Your proposal means that the person writing the lockedIncrement(), which is a
> perfectly reasonable thing to do, simply cannot write it in a way that has a
> @safe interface
Correct, the rules of my proposal apply to lockedIncrement(). They
apply to `shared` generally.
lockedIncrement() is not a threadsafe function. You can't call it on a
shared instance, because `int`s API (ie, all intrinsic operations) are
not threadsafe.
lockedIncrement() can't promise threadsafe access to `shared(int)*`,
so the argument is not shared.
Your program made the correct compile error about doing unsafety, but
the location of the compile error is different under my proposal;
complexity is worn by the shared library author, rather
Re: shared - i need it to be useful
On Mon, Oct 22, 2018 at 12:50 AM Walter Bright via Digitalmars-d wrote: > > On 10/21/2018 5:54 PM, Manu wrote: > > Would you please respond to my messages, and specifically, respond to > > the code that I presented to you in response to your broken example. > > Or any of my earlier fragments throughout this thread. I've shared > > quite a few, and so far, nobody has ever produced a criticism of any > > of my fragments. They've just been skipped over. > > That's just the problem. You've posted 62 messages so far in this thread, and > then there's all the ones Nicholas posted. I sent it twice... again just a short while ago right before this one... but you responded to this one and not that one O_o > Trying to assemble the "earlier fragments throughout this thread" is not > practical for readers, and the endless nature of this thread is ample evidence > for it. The n.g. is a place to discuss a proposal, not the proposal itself. > > This change is definitely merits an actual proposal DIP, so that one is > assured > of seeing the complete proposal, rationale, examples, etc., in one document, > as > well as not being distracted by sidebars, thread drift, and mistakes. This > document can evolve with corrections and clarifications from the discussion, > and > anyone can get up to speed quickly by just reading the latest version of it. Okay, but I still want you to respond to my corrections of your program, which were in direct response to you... twice. > > But the one aimed directly at your own most recent sample program > > addresses your program directly. > > My most recent sample program was a direct criticism of one of your fragments, > so please don't say "nobody has ever ...". I do understand your frustration at > finding it hard to get your point across, but the problem at least for me is > trying to mine it from nuggets scattered across 62 posts. Mine it, refine it, > cast it into an ingot, then present it as a DIP. I posted it, twice... 2 messages, back to back, and you're responding to this one, and not that one. I'll post it again...
Re: shared - i need it to be useful
On 10/21/2018 11:58 AM, Timon Gehr wrote: [...] Thank you, Timon, for a nice explanation of what I was trying to express.
Re: shared - i need it to be useful
On 10/21/2018 5:54 PM, Manu wrote: Would you please respond to my messages, and specifically, respond to the code that I presented to you in response to your broken example. Or any of my earlier fragments throughout this thread. I've shared quite a few, and so far, nobody has ever produced a criticism of any of my fragments. They've just been skipped over. That's just the problem. You've posted 62 messages so far in this thread, and then there's all the ones Nicholas posted. Trying to assemble the "earlier fragments throughout this thread" is not practical for readers, and the endless nature of this thread is ample evidence for it. The n.g. is a place to discuss a proposal, not the proposal itself. This change is definitely merits an actual proposal DIP, so that one is assured of seeing the complete proposal, rationale, examples, etc., in one document, as well as not being distracted by sidebars, thread drift, and mistakes. This document can evolve with corrections and clarifications from the discussion, and anyone can get up to speed quickly by just reading the latest version of it. > But the one aimed directly at your own most recent sample program > addresses your program directly. My most recent sample program was a direct criticism of one of your fragments, so please don't say "nobody has ever ...". I do understand your frustration at finding it hard to get your point across, but the problem at least for me is trying to mine it from nuggets scattered across 62 posts. Mine it, refine it, cast it into an ingot, then present it as a DIP.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:22:19 UTC, Manu wrote: On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d wrote: On 10/21/2018 2:08 PM, Walter Bright wrote: > On 10/21/2018 12:20 PM, Nicholas Wilson wrote: >> Yes, but the problem you describe is arises from implicit >> conversion in the other direction, which is not part of the >> proposal. > > It's Manu's example. Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text, No no, they're repeated, not scattered, because I seem to have to keep repeating it over and over, because nobody is reading the text, or perhaps imaging there is a lot more text than there is. I told you this is what happens with forum posts 4 days ago, yet you didn't listen: https://forum.dlang.org/post/fokdcnzircoiuhrhz...@forum.dlang.org opinions, and handwavy stuff. You mean like every post in opposition which disregards the rules and baselessly asserts it's a terrible idea? :/ There's nothing to point to that is "the proposal". You can go back to the OP, not a single detail is changed at any point, but I've repeated it a whole bunch of times (including in direct response to your last post) and the summary has become more concise, but not different. 1. Shared has no read or write access to data 2. Functions with shared arguments are threadsafe with respect to those arguments a. This is a commitment that must be true in _absolute terms_ (there exists discussion about the ways that neighbours must not undermine this promise) b. There are numerous examples demonstrating how to configure this (TL;DR: use encapsulation, and @trusted at the bottom of the stack) If you can find a legitimate example where those rules don't hold, I want to see it. But every example so far has been based on a faulty premise where those 2 simple rules were not actually applied. Put it all together in a 2-3 page proposal elsewhere, so he doesn't have to hunt everything out in a blizzard of forum posts. I responded to your faulty program directly with the correct program, and you haven't acknowledged it. Did you see it? I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior. I have written this program a couple of times, including in direct response to your last sample program. You seem to have dismissed it... where is your response to that program, or my last entire post? For examples of how to do it: https://github.com/dlang/DIPs/tree/master/DIPs Trying to rewrite the semantics of shared is not a simple task, doing multithreading correctly is a minefield of "OOPS! I didn't think of that!" and if anything cries out for a DIP, your and Manu's proposal does. Yes, I agree it's DIP worthy. But given the almost nothing but overt hostility I've received here, why on earth would I waste my time writing a DIP? I put months into my other DIP which sits gathering dust... if this thread inspired any confidence that it would be well-received I would make the effort, but the critical reception we've seen here is... a bit strange. It's a 2-point proposal, the rules are **SO SIMPLE**, which is why I love it. How it can be misunderstood is something I'm having trouble understanding, and I don't know how to make it any clearer than I already have; numerous times over, including in my last reply to you, which you have ignored and dismissed it seems. Please go back and read my response to your last program. He did not say to write a full DIP, just a proposal, so he knows exactly what you mean, just as I said. It will require a DIP eventually, but he didn't ask you to write one now.
Re: shared - i need it to be useful
On Sun, 21 Oct 2018 17:35:38 -0700, Manu wrote: > On Sun, Oct 21, 2018 at 3:15 PM Neia Neutuladh via Digitalmars-d > wrote: >> If we only used your proposal and only used @safe code, we wouldn't >> have any data races, but that's only because we wouldn't have any >> shared data. We'd have shared *variables*, but they would not contain >> any data we could read or alter, and that's pretty much useless. > > I've shown an implementation of Atomic(T) 3 times now... no response any > time. Why is it being dismissed? Do I need to write it more times? > This is a clear demonstration of how to build the foundation of the > @safe threadsafe stack. Yes, Atomic can be implemented here as @trusted code, and maybe using @safe compiler intrinsics for some situations. That's useful! But it's not *enough*. It would require a lot of work to refit existing code to using only atomic operations, and it would end up looking rather clunky a lot of the time. If you're doing anything complex with a complex object graph, you're going to have a terrible time. You need to copy that object graph (atomically), which is going to be expensive and provides non-trivial restrictions on what you can store. So I'm not keen on a world in which all multithreading is using atomic structs. Unless, when you were talking about Atomic, you actually meant a wrapper containing some sort of lock, allowing you to submit arbitrary delegates to mutate the data within in a serialized way, similar to Atila Neves's fearless library. Which really stretches the definition of "atomic". >> Currently, it helps because casting unshared to shared is not @safe, >> because it makes it trivial to get multiple threads with unshared >> references to the same data. > > No no, that's a massive smell. That means anytime anyone wants to > distribute something, they need to perform unsafe casts. That's not > okay. Casting thread-local to shared makes it easy to cause errors, and that's why it's a massive smell. Making it silent doesn't eliminate the smell. > 100% of my SMP code works with my proposal, and something close to 0% > works with shared as it is today. (assuming we desire @safe interaction, > which we do, because threading is hard enough already!) You want un-shared things to implicitly cast to shared. You don't want to have to allocate anything as shared, and you do want to pass absolutely anything to any thread. You can write a @trusted assumeShared function, analogous to assumeUnique, to accomplish that. It would be slightly more awkward than what you're proposing, but it accomplishes one of your two goals: convert non-shared things to shared things in @safe code. The other goal in your proposal is for some code that currently compiles not to. So you should be able to write the code you want, albeit with an increased risk of bugs. Or you could write a template that wraps a shared thing and forbids field access and assignment. >> And that's when you're using shared as expected rather than doing >> something weird. > > No, I *expect* to use shared in @safe code, and not write any unsafe > code ever. shared doesn't model a useful interaction now, not in any > way. > > Today, access to shared data members are unrestricted and completely > unsafe Yes, and that's bad and should be changed. > passing data into something like a parallel-for requires unsafe > casts. Or allocating data as shared, which is the recommended way, because that makes absolutely certain that, from the start, no code has an un-shared copy of that data. No casts needed.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:55:00 UTC, Timon Gehr wrote: On 22.10.18 02:46, Nicholas Wilson wrote: On Monday, 22 October 2018 at 00:38:33 UTC, Timon Gehr wrote: I just did, Link please? https://forum.dlang.org/post/pqii8k$11u3$1...@digitalmars.com That contains no code. Not all of the parties that participate in the data race are in @trusted code. There are two cases of @trusted code, bad and good. Bad trusted code can indeed be misused by @safe code to corrupt memory. Good trusted code cannot. What part of the proposal breaks the type system? The implicit conversion to shared implies that the passed to function is @safe iff all the functions it calls are @safe only call function with that parameter to other shared @safe functions _OR_ they are @trusted. The point of @trusted is modularity: you manually check @trusted code according to some set of restrictions and then you are sure that there is no memory corruption. Yes. And?
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 5:55 PM Timon Gehr via Digitalmars-d wrote: > > On 22.10.18 02:45, Manu wrote: > > On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d > > wrote: > >> > >> On 21.10.18 20:46, Manu wrote: > Shared data is only useful if, at some point, it is read/written, > presumably by > casting it to unshared in @trusted code. As soon as that is done, you've > got a > data race with the other existing unshared aliases. > >>> If such a race is possible, then the @trusted function is not > >>> threadsafe, so it is not @trusted by definition. > >>> You wrote a bad @trusted function, and you should feel bad. > >>> ... > >> > >> I wonder where this "each piece of code is maintained by only one person > >> and furthermore this is the only person that will suffer if the code has > >> bugs" mentality comes from. It is very popular as well as obviously > >> nonsense. > >> > >>> The simplest way to guarantee that no unsafe access is possible is to > >>> use encapsulation to assure no unregulated access exists. > >> > >> This only works if untrusted programmers (i.e. programmers who are only > >> allowed to write/modify @safe code) are not allowed to change your > >> class. I.e. it does not work. > > > > Have you ever cracked open std::map and 'fixed' it because you thought > > it was bad? > > Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic > > without understanding what they're doing. > > > > You are not proposing to let core.atomic.Atomic convert to shared > implicitly, you are proposing to do that for all classes. You can always implicitly convert to shared. Where did I ever say anything like that? I'm sure I've never said this. How do these transformations of what I've said keep happening? > > You seem to be stuck on the detail whether you can trust the @trusted > > author though... > > Again: the @safe author is the problem. I don't follow. The @safe author is incapable of doing threadsafety violation. They can only combine threadsafe functions. They can certainly produce a program that doesn't work, and they are capable of ordering issues, but that's not the same as data-race related crash bugs.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 5:40 PM Timon Gehr via Digitalmars-d wrote: > > On 21.10.18 21:04, Manu wrote: > > On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d > > wrote: > >> > >> On 21.10.18 17:54, Nicholas Wilson wrote: > >>> > As soon as that is done, you've got a data race with the other > existing unshared aliases. > >>> > >>> You're in @trusted code, that is the whole point. The onus is on the > >>> programmer to make that correct, same with regular @safe/@trusted@system > >>> code. > >> > >> Not all of the parties that participate in the data race are in @trusted > >> code. The point of @trusted is modularity: you manually check @trusted > >> code according to some set of restrictions and then you are sure that > >> there is no memory corruption. > >> > >> Note that you are not allowed to look at any of the @safe code while > >> checking your @trusted code. You will only see an opaque interface to > >> the @safe code that you call and all you know is that all the @safe code > >> type checks according to @safe rules. Note that there might be an > >> arbitrary number of @safe functions and methods that you do not see. > >> > >> Think about it this way: you first write all the @trusted and @system > >> code, and some evil guy who does not like you comes in after you looks > >> at your code and writes all the @safe code. If there is any memory > >> corruption, it will be your fault and you will face harsh consequences. > >> Now, design the @safe type checking rules. It won't be MP! > >> > >> Note that there may well be a good way to get the good properties of MP > >> without breaking the type system, but MP itself is not good because it > >> breaks @safe. > > > > Show me. Nobody has been able to show that yet. I'd really like to know > > this. > > > > I just did, There's no code there... just a presumption that the person who wrote the @trusted code did not deliver the promise they made. > but if you really need to, give me a non-trivial piece of> correct > multithreaded code that accesses some declared-unshared field > from a shared method and I will show you how the evil guy would modify > some @safe code in it and introduce race conditions. It needs to be your > code, as otherwise you will just claim again that it is me who wrote bad > @trusted code. You can pick on any of my prior code fragments. They've all been ignored so far.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:46:04 UTC, Walter Bright wrote: That's what I was referring to, and Manu's example. It doesn't work, as I pointed out. I'm pretty sure it does, but please repeat it. We will eventually. This started as a "please point out any problems with this" and has probably outlived that phase. You'll need to address the issues raised here in the DIP. That is a given. You would do well to heed it for your own DIPs.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 5:50 PM Walter Bright via Digitalmars-d wrote: > > On 10/21/2018 4:12 PM, Nicholas Wilson wrote: > > On Sunday, 21 October 2018 at 21:32:14 UTC, Walter Bright wrote: > >> On 10/21/2018 2:08 PM, Walter Bright wrote: > >>> On 10/21/2018 12:20 PM, Nicholas Wilson wrote: > Yes, but the problem you describe is arises from implicit conversion in > the > other direction, which is not part of the proposal. > >>> > >>> It's Manu's example. > >> > >> Then I don't know what the proposal is. Pieces of it appear to be scattered > >> over numerous posts, mixed in with other text, opinions, and handwavy > >> stuff. > >> There's nothing to point to that is "the proposal". > > > > The proposal is: > > > > Implicit conversion _to_ shared, e.g. passing it to a thread entry point, > > and > > not implicit conversion _from_ shared (just like implicit const > > conversions). > > That's what I was referring to, and Manu's example. It doesn't work, as I > pointed out. > > > >> I suggest you and Manu write up a proper proposal. Something that is > >> complete, > >> has nothing else in it, has a rationale, illuminating examples, and > >> explains > >> why alternatives are inferior. > > > > We will eventually. This started as a "please point out any problems with > > this" > > and has probably outlived that phase. > > You'll need to address the issues raised here in the DIP. Would you please respond to my messages, and specifically, respond to the code that I presented to you in response to your broken example. Or any of my earlier fragments throughout this thread. I've shared quite a few, and so far, nobody has ever produced a criticism of any of my fragments. They've just been skipped over. But the one aimed directly at your own most recent sample program addresses your program directly.
Re: shared - i need it to be useful
On 22.10.18 02:46, Nicholas Wilson wrote: On Monday, 22 October 2018 at 00:38:33 UTC, Timon Gehr wrote: I just did, Link please? https://forum.dlang.org/post/pqii8k$11u3$1...@digitalmars.com
Re: shared - i need it to be useful
On 22.10.18 02:45, Manu wrote: On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d wrote: On 21.10.18 20:46, Manu wrote: Shared data is only useful if, at some point, it is read/written, presumably by casting it to unshared in @trusted code. As soon as that is done, you've got a data race with the other existing unshared aliases. If such a race is possible, then the @trusted function is not threadsafe, so it is not @trusted by definition. You wrote a bad @trusted function, and you should feel bad. ... I wonder where this "each piece of code is maintained by only one person and furthermore this is the only person that will suffer if the code has bugs" mentality comes from. It is very popular as well as obviously nonsense. The simplest way to guarantee that no unsafe access is possible is to use encapsulation to assure no unregulated access exists. This only works if untrusted programmers (i.e. programmers who are only allowed to write/modify @safe code) are not allowed to change your class. I.e. it does not work. Have you ever cracked open std::map and 'fixed' it because you thought it was bad? Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic without understanding what they're doing. You are not proposing to let core.atomic.Atomic convert to shared implicitly, you are proposing to do that for all classes. You seem to be stuck on the detail whether you can trust the @trusted author though... Again: the @safe author is the problem.
Re: shared - i need it to be useful
On 10/21/2018 4:12 PM, Nicholas Wilson wrote: On Sunday, 21 October 2018 at 21:32:14 UTC, Walter Bright wrote: On 10/21/2018 2:08 PM, Walter Bright wrote: On 10/21/2018 12:20 PM, Nicholas Wilson wrote: Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal. It's Manu's example. Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text, opinions, and handwavy stuff. There's nothing to point to that is "the proposal". The proposal is: Implicit conversion _to_ shared, e.g. passing it to a thread entry point, and not implicit conversion _from_ shared (just like implicit const conversions). That's what I was referring to, and Manu's example. It doesn't work, as I pointed out. I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior. We will eventually. This started as a "please point out any problems with this" and has probably outlived that phase. You'll need to address the issues raised here in the DIP.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:38:33 UTC, Timon Gehr wrote: I just did, Link please?
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 5:35 PM Timon Gehr via Digitalmars-d wrote: > > On 21.10.18 20:46, Manu wrote: > >> Shared data is only useful if, at some point, it is read/written, > >> presumably by > >> casting it to unshared in @trusted code. As soon as that is done, you've > >> got a > >> data race with the other existing unshared aliases. > > If such a race is possible, then the @trusted function is not > > threadsafe, so it is not @trusted by definition. > > You wrote a bad @trusted function, and you should feel bad. > > ... > > I wonder where this "each piece of code is maintained by only one person > and furthermore this is the only person that will suffer if the code has > bugs" mentality comes from. It is very popular as well as obviously > nonsense. > > > The simplest way to guarantee that no unsafe access is possible is to > > use encapsulation to assure no unregulated access exists. > > This only works if untrusted programmers (i.e. programmers who are only > allowed to write/modify @safe code) are not allowed to change your > class. I.e. it does not work. Have you ever cracked open std::map and 'fixed' it because you thought it was bad? Of course not. Same applies here. Nobody 'fixes' core.atomic.Atomic without understanding what they're doing. You seem to be stuck on the detail whether you can trust the @trusted author though... that is a reasonable point of debate, but it's a slightly separate topic. I am confident that the number of @trusted functions required to found a useful stack are low, probably countable with fingers, and always in a library. If we can put aside that point of debate just for now; whether you feel the @trusted author can be trusted, assuming that they can, does the model work? Can you break the model as I have presented it? If not; if the model is sound, then we can begin the discussion you're alluding to and talk about opportunities to improve on static guarantees for @trusted authors, or ways to communicate their responsibility clearly, and patterns to assure success.
Re: shared - i need it to be useful
On Monday, 22 October 2018 at 00:32:35 UTC, Timon Gehr wrote: This only works if untrusted programmers (i.e. programmers who are only allowed to write/modify @safe code) are not allowed to change your class. I.e. it does not work. This is the basis of the current @safe/@trusted/@system model. Are you saying it is useless?
Re: shared - i need it to be useful
On 21.10.18 21:04, Manu wrote: On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d wrote: On 21.10.18 17:54, Nicholas Wilson wrote: As soon as that is done, you've got a data race with the other existing unshared aliases. You're in @trusted code, that is the whole point. The onus is on the programmer to make that correct, same with regular @safe/@trusted@system code. Not all of the parties that participate in the data race are in @trusted code. The point of @trusted is modularity: you manually check @trusted code according to some set of restrictions and then you are sure that there is no memory corruption. Note that you are not allowed to look at any of the @safe code while checking your @trusted code. You will only see an opaque interface to the @safe code that you call and all you know is that all the @safe code type checks according to @safe rules. Note that there might be an arbitrary number of @safe functions and methods that you do not see. Think about it this way: you first write all the @trusted and @system code, and some evil guy who does not like you comes in after you looks at your code and writes all the @safe code. If there is any memory corruption, it will be your fault and you will face harsh consequences. Now, design the @safe type checking rules. It won't be MP! Note that there may well be a good way to get the good properties of MP without breaking the type system, but MP itself is not good because it breaks @safe. Show me. Nobody has been able to show that yet. I'd really like to know this. I just did, but if you really need to, give me a non-trivial piece of correct multithreaded code that accesses some declared-unshared field from a shared method and I will show you how the evil guy would modify some @safe code in it and introduce race conditions. It needs to be your code, as otherwise you will just claim again that it is me who wrote bad @trusted code.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 3:15 PM Neia Neutuladh via Digitalmars-d wrote: > > On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote: > > On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d > > wrote: > >> Note that there may well be a good way to get the good properties of MP > >> without breaking the type system, but MP itself is not good because it > >> breaks @safe. > > > > Show me. Nobody has been able to show that yet. I'd really like to know > > this. > > If we only used your proposal and only used @safe code, we wouldn't have > any data races, but that's only because we wouldn't have any shared data. > We'd have shared *variables*, but they would not contain any data we could > read or alter, and that's pretty much useless. I've shown an implementation of Atomic(T) 3 times now... no response any time. Why is it being dismissed? Do I need to write it more times? This is a clear demonstration of how to build the foundation of the @safe threadsafe stack. > To use your proposal, we need to cast data back from shared to unshared. > When it's unshared, we need to make sure that exactly one thread has a > reference to that data as unshared. No, you just need to make sure that access is atomic or synchronised in a proper way, and if you do cast away shared in some low-level @trusted function, make sure that reference doesn't escape. You can do it, I have faith. > And @safe *should* help us with that. Totally. > Currently, it helps because casting unshared to shared is not @safe, > because it makes it trivial to get multiple threads with unshared > references to the same data. No no, that's a massive smell. That means anytime anyone wants to distribute something, they need to perform unsafe casts. That's not okay. Modeling shared-ness/unshared-ness is not *useful* in any way that I have been able to identify. Modelling what it means to be threadsafe is useful in every application I've ever written. 100% of my SMP code works with my proposal, and something close to 0% works with shared as it is today. (assuming we desire @safe interaction, which we do, because threading is hard enough already!) > And that's when you're using shared as > expected rather than doing something weird. No, I *expect* to use shared in @safe code, and not write any unsafe code ever. shared doesn't model a useful interaction now, not in any way. Today, access to shared data members are unrestricted and completely unsafe, passing data into something like a parallel-for requires unsafe casts.
Re: shared - i need it to be useful
On 21.10.18 20:46, Manu wrote: Shared data is only useful if, at some point, it is read/written, presumably by casting it to unshared in @trusted code. As soon as that is done, you've got a data race with the other existing unshared aliases. If such a race is possible, then the @trusted function is not threadsafe, so it is not @trusted by definition. You wrote a bad @trusted function, and you should feel bad. ... I wonder where this "each piece of code is maintained by only one person and furthermore this is the only person that will suffer if the code has bugs" mentality comes from. It is very popular as well as obviously nonsense. The simplest way to guarantee that no unsafe access is possible is to use encapsulation to assure no unregulated access exists. This only works if untrusted programmers (i.e. programmers who are only allowed to write/modify @safe code) are not allowed to change your class. I.e. it does not work.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 2:35 PM Walter Bright via Digitalmars-d wrote: > > On 10/21/2018 2:08 PM, Walter Bright wrote: > > On 10/21/2018 12:20 PM, Nicholas Wilson wrote: > >> Yes, but the problem you describe is arises from implicit conversion in the > >> other direction, which is not part of the proposal. > > > > It's Manu's example. > > Then I don't know what the proposal is. Pieces of it appear to be scattered > over > numerous posts, mixed in with other text, No no, they're repeated, not scattered, because I seem to have to keep repeating it over and over, because nobody is reading the text, or perhaps imaging there is a lot more text than there is. > opinions, and handwavy stuff. You mean like every post in opposition which disregards the rules and baselessly asserts it's a terrible idea? :/ > There's nothing to point to that is "the proposal". You can go back to the OP, not a single detail is changed at any point, but I've repeated it a whole bunch of times (including in direct response to your last post) and the summary has become more concise, but not different. 1. Shared has no read or write access to data 2. Functions with shared arguments are threadsafe with respect to those arguments a. This is a commitment that must be true in _absolute terms_ (there exists discussion about the ways that neighbours must not undermine this promise) b. There are numerous examples demonstrating how to configure this (TL;DR: use encapsulation, and @trusted at the bottom of the stack) If you can find a legitimate example where those rules don't hold, I want to see it. But every example so far has been based on a faulty premise where those 2 simple rules were not actually applied. I responded to your faulty program directly with the correct program, and you haven't acknowledged it. Did you see it? > I suggest you and Manu write up a proper proposal. Something that is complete, > has nothing else in it, has a rationale, illuminating examples, and explains > why > alternatives are inferior. I have written this program a couple of times, including in direct response to your last sample program. You seem to have dismissed it... where is your response to that program, or my last entire post? > For examples of how to do it: > > https://github.com/dlang/DIPs/tree/master/DIPs > > Trying to rewrite the semantics of shared is not a simple task, doing > multithreading correctly is a minefield of "OOPS! I didn't think of that!" and > if anything cries out for a DIP, your and Manu's proposal does. Yes, I agree it's DIP worthy. But given the almost nothing but overt hostility I've received here, why on earth would I waste my time writing a DIP? I put months into my other DIP which sits gathering dust... if this thread inspired any confidence that it would be well-received I would make the effort, but the critical reception we've seen here is... a bit strange. It's a 2-point proposal, the rules are **SO SIMPLE**, which is why I love it. How it can be misunderstood is something I'm having trouble understanding, and I don't know how to make it any clearer than I already have; numerous times over, including in my last reply to you, which you have ignored and dismissed it seems. Please go back and read my response to your last program.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 11:31 AM Manu wrote:
>
> On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d,
> wrote:
> >
> > On 10/20/2018 11:24 AM, Manu wrote:
> > > This is an unfair dismissal.
> >
> > It has nothing at all to do with fairness. It is about what the type system
> > guarantees in @safe code. To repeat, the current type system guarantees in
> > @safe
> > code that T* and shared(T)* do not point to the same memory location.
> >
> > Does your proposal maintain that or not? It's a binary question.
>
> By the definition Nick pulled from Wikipedia and posted for you a few
> posts back, yes, my proposal satisfies Wikipedia's definition of no
> aliasing. I understand that property is critical, and I have carefully
> designed for it.
>
> > > I'm not sure you've understood the proposal.
> > > This is the reason for the implicit conversion. It provides safe
> > > transition.
> >
> > I don't see any way to make an implicit T* to shared(T)* safe, or vice
> > versa.
> > The T* code can create more aliases that the conversion doesn't know about,
> > and
> > the shared(T)* code can hand out aliases to other threads. So it all falls
> > to
> > pieces.
>
> T* can't make additional T* aliases on other threads; there can only
> be one thread with T*.
> shared(T)* can not make a T*.
> shared(T)* has no read or write access, so it's not an alias of T* by
> Wikipedia's definition.
>
> Only threadsafe functions can do anything to T.
> The leap of faith is; some @trusted utility functions at the bottom of
> the shared stack makes a promise that it is threadsafe, and must
> deliver that promise.
> I don't think this is unreasonable; this is the nature of @trusted
> functions, they make a promise, and they must keep it.
> If the trusted function does not lie, then the chain of trust holds
> upwards through the stack.
>
> The are very few such trusted functions in practise. Like, similar to
> the number of digits you have.
>
> > Using a 'scope' qualifier won't work, because 'scope' isn't transitive,
> > while shared is, i.e. U** and shared(U*)*.
>
> I don't think I depend on scope in any way.
> That was an earlier revision of thinking in an older thread.
>
> > > I'm not sure how to clarify it, what can I give you?
> >
> > Write a piece of code that does such an implicit conversion that you argue
> > is
> > @safe. Make the code as small as possible. Your example:
> >
> > > int* a;
> > > shared(int)* b = a;
> >
> > This is not safe.
> >
> > Manu's Proposal ---
> > @safe:
> > int i;
> > int* a =
> > StartNewThread(a); // Compiles! Coder has no idea!
> >
> > ... in the new thread ...
> > void StartOfNewThread(shared(int)* b) {
> >
> > ... we have two threads accessing 'i',
> > one thinks it is shared, the other unshared,
> > and StartOfNewThread() has no idea and anyone
> > writing code for StartOfNewThread() has no way
> > to know anything is wrong ...
> >
> > lockedIncrement(b); // Data Race!
> > }
>
> This program doesn't compile. You receive an error because it is not safe.
> The function is `lockedIncrement(int*)`. It can't receive a shared
> argument; the function is not threadsafe by my definition.
> You have written a program that produces the expected error that
> alerts you that you have tried to do un-@safe and make a race.
>
> Stanislav produced this same program, and I responded with the correct
> program a few posts back.
> I'll repeat it here; the @safe program to model this interaction is:
>
> @safe:
>
> // function is NOT threadsafe by my definition, can not be called on
> shared arguments
> void atomicIncrement(int*);
>
> struct Atomic(T)
> {
> // encapsulare the unsafe data so it's inaccessible by any unsafe means
> private T val;
>
> // perform the unsafe cast in a trusted function
> // we are able to assure a valid calling context by encapsulating
> the data above
> void opUnary(string op : "++")() shared @trusted {
> atomicIncrement(cast(T*)); }
> }
>
> Atomic!int i;
> Atomic!int* a =
> StartNewThread(a); // Compiles, of course!
> ++i; // no race
>
> ... in the new thread ...
> void StartOfNewThread(shared(Atomic!int)* b) {
> ... we have two threads accessing 'i', one has thread-local access,
> this one has a restricted shared access.
> here, we have a shared instance, so we can only access `b` via
> threadsafe functions.
> as such, we can manipulate `b` without fear.
> ++i; // no race!
> }
>
>
> > Your proposal means that the person writing the lockedIncrement(), which is
> > a
> > perfectly reasonable thing to do, simply cannot write it in a way that has a
> > @safe interface
>
> Correct, the rules of my proposal apply to lockedIncrement(). They
> apply to `shared` generally.
> lockedIncrement() is not a threadsafe function. You can't call it on a
> shared instance, because `int`s API (ie, all intrinsic operations) are
> not threadsafe.
> lockedIncrement() can't promise threadsafe access to `shared(int)*`,
> so the argument
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 22:12:18 UTC, Neia Neutuladh wrote: On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote: On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d wrote: Note that there may well be a good way to get the good properties of MP without breaking the type system, but MP itself is not good because it breaks @safe. Show me. Nobody has been able to show that yet. I'd really like to know this. If we only used your proposal and only used @safe code, we wouldn't have any data races, Only of the @trusted implementation is thread safe, which it _should_ be. This is the same caveat as non threaded -@safe/@tusted code. but that's only because we wouldn't have any shared data. We'd have shared *variables*, but they would not contain any data we could read Reads must be atomic. or alter, and that's pretty much useless. To use your proposal, we need to cast data back from shared to unshared. Yes but this is in the @trusted implementation that forms the basis of your threadsafety. When it's unshared, we need to make sure that exactly one thread has a reference to that data as unshared. Nod. And @safe *should* help us with that. Nod. Currently, it helps because casting unshared to shared is not @safe, This remains the case, and should be done (enforced by the compiler) only in @trusted/@system code as a basis for thread safe, @safe code. because it makes it trivial to get multiple threads with unshared references to the same data. That is @trusted or @system code and therefore is the programmers responsibility. And that's when you're using shared as expected rather than doing something weird. That forms the basis of your thread safe stack. From there on, the basis that shared arguments to functions are treated safely in the presence of threading means that code that calls the @trusted implementations is @safe.
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 21:32:14 UTC, Walter Bright wrote: On 10/21/2018 2:08 PM, Walter Bright wrote: On 10/21/2018 12:20 PM, Nicholas Wilson wrote: Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal. It's Manu's example. Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text, opinions, and handwavy stuff. There's nothing to point to that is "the proposal". The proposal is: Implicit conversion _to_ shared, e.g. passing it to a thread entry point, and not implicit conversion _from_ shared (just like implicit const conversions). Shared disables reads and writes Your confusion results from the use of atomic add which, in Manu's examples had a different signature than before. I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior. We will eventually. This started as a "please point out any problems with this" and has probably outlived that phase. Trying to rewrite the semantics of shared is not a simple task, Not as much as trying to explain it! Having talked to Manu in person it is much easier to understand. doing multithreading correctly is a minefield of "OOPS! I didn't think of that!" The above case in point, this is about assuming your implementation of thread safe primitives are thread safe (@trusted) that you use it correctly (@safe).
Re: shared - i need it to be useful
On Sun, 21 Oct 2018 12:04:16 -0700, Manu wrote: > On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d > wrote: >> Note that there may well be a good way to get the good properties of MP >> without breaking the type system, but MP itself is not good because it >> breaks @safe. > > Show me. Nobody has been able to show that yet. I'd really like to know > this. If we only used your proposal and only used @safe code, we wouldn't have any data races, but that's only because we wouldn't have any shared data. We'd have shared *variables*, but they would not contain any data we could read or alter, and that's pretty much useless. To use your proposal, we need to cast data back from shared to unshared. When it's unshared, we need to make sure that exactly one thread has a reference to that data as unshared. And @safe *should* help us with that. Currently, it helps because casting unshared to shared is not @safe, because it makes it trivial to get multiple threads with unshared references to the same data. And that's when you're using shared as expected rather than doing something weird.
Re: shared - i need it to be useful
On 10/21/2018 2:08 PM, Walter Bright wrote: On 10/21/2018 12:20 PM, Nicholas Wilson wrote: Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal. It's Manu's example. Then I don't know what the proposal is. Pieces of it appear to be scattered over numerous posts, mixed in with other text, opinions, and handwavy stuff. There's nothing to point to that is "the proposal". I suggest you and Manu write up a proper proposal. Something that is complete, has nothing else in it, has a rationale, illuminating examples, and explains why alternatives are inferior. For examples of how to do it: https://github.com/dlang/DIPs/tree/master/DIPs Trying to rewrite the semantics of shared is not a simple task, doing multithreading correctly is a minefield of "OOPS! I didn't think of that!" and if anything cries out for a DIP, your and Manu's proposal does.
Re: shared - i need it to be useful
On 10/21/2018 12:20 PM, Nicholas Wilson wrote: Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal. It's Manu's example.
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 19:22:45 UTC, Manu wrote: On Sun, Oct 21, 2018 at 5:50 AM Stanislav Blinov via Digitalmars-d wrote: Because the whole reason to have `shared` is to avoid the extraneous checks that you mentioned above, No, it is to assure that you write correct not-broken code. You can do that without `shared`. and only write actual useful code (i.e. lock-write-unlock, or read-put-to-queue-repeat, or whatever), not busy-work (testing if the file is open on every call). `shared` is no comment on performance. You have written a slow locking API. If you care about perf, you would write a completely different API that favours perf. This not impossible, nor even particularly hard. You're conflating your assumptions about the code with the topic of this discussion. I can write a million examples and you'll still find a million reasons to talk about how they're incorrectly implemented, instead of focusing on merits or disadvantages of your proposal with the code given as is. If you have a `shared` reference, it better be to existing data. I mean, if I dereference a pointer, it had better not be null! Why would you share a null pointer? That's why having `shared` and un-`shared` references to the same data simultaneously is not safe: you can't guarantee in any way that the owning thread doesn't invalidate the data through it's non-`shared` reference while you're doing your threadsafe `shared` work; you can only "promise" that by convention (documentation). The owning thread is not a special actor. Your reasoning is wonky here. Why have it then at all? If it's not a "special" actor, just make all shared data `shared`. But your proposal specifically targets the conversion, suggesting you *do* need a special actor. And I have partially-read or partially-written data. I expect you flushed before killing the file. So? The threads still weren't done yet. Or Maybe I call closeFile(), main thread continues and opens another file, which gives the same file descriptor, `shared` references to FileHandle which the user forgot to wait on continue to work oblivious to the fact that it's a different file now. It's wild to suggest that ANY design for `shared` should somehow deal with the OS recycling a file handle... I'm not suggesting that at all, you've completely misrepresenting what I'm saying by splitting a quote. And it's still not an un-@safe crash! It's just a program with a bug. Ok, if you say that sticking @safe on code that can partially piece together data from unrelated sources is fine, then sure. > I'm going to assume that `shareWithThreads()` was implemented > by an 'expert' who checked the function results for errors... But you can only find out about these errors in `waitForThreads`, the very call that the user "forgot" to make! ...what? Please suggest another way of handling errors reported by other threads. More `shared` state? [ ... snip ... ] You have to concede defeat at this point. I agree. No matter how hard I try or how many times I ask you to demonstrate, I still fail to see the value in assuming @safe implicitly conversion of mutable data to shared. Instead of defending your proposal, you chose to attack the opposition. You've defeated me, flawless victory. Destroy my proposal with another legitimately faulty program. Is there a point? I post code, you: "nah, that's wrong". Steven posts code, you: "nah, that's wrong". Timon posts code, you: "nah, that's wrong". Walter posts code, you: "nah, that's wrong"... What's right then?
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 19:07:37 UTC, Nicholas Wilson wrote: On Sunday, 21 October 2018 at 09:50:09 UTC, Walter Bright wrote: Your proposal makes that impossible because the compiler would allow unshared data to be implicitly typed as shared. Yes, but not the other way around. Whoops that should read Your proposal makes that impossible No because the compiler would allow unshared data to be implicitly typed as shared. Yes, but the problem you describe is arises from implicit conversion in the other direction, which is not part of the proposal.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 5:50 AM Stanislav Blinov via Digitalmars-d wrote: > > On Sunday, 21 October 2018 at 05:47:14 UTC, Manu wrote: > > >> And yet, if we're lucky, we get > >> a consistent instacrash. If we're unlucky, we get memory > >> corruption, or an unsolicited write to another currently open > >> file, either of which can go unnoticed for some time. > > > Woah! Now this is way off-piste.. > > Why would get a crash? Why would get memory corruption? None of > > those things make sense. > > Because the whole reason to have `shared` is to avoid the > extraneous checks that you mentioned above, No, it is to assure that you write correct not-broken code. > and only write actual > useful code (i.e. lock-write-unlock, or read-put-to-queue-repeat, > or whatever), not busy-work (testing if the file is open on every > call). `shared` is no comment on performance. You have written a slow locking API. If you care about perf, you would write a completely different API that favours perf. This not impossible, nor even particularly hard. > If you have a `shared` reference, it better be to existing > data. I mean, if I dereference a pointer, it had better not be null! > If it isn't, the program is invalid already: you've shared > something that doesn't "exist" (good for marketing, not so good > for multithreading). I mean, if I dereference a pointer, it had better not be null! > That's why having `shared` and un-`shared` > references to the same data simultaneously is not safe: you can't > guarantee in any way that the owning thread doesn't invalidate > the data through it's non-`shared` reference while you're doing > your threadsafe `shared` work; you can only "promise" that by > convention (documentation). The owning thread is not a special actor. Your reasoning is wonky here. Lets say there is no owning instance, only a collection of shared instances... any one of them can theoretically do an operation that interferes with the others. That's issues for general threading, and no design for `shared` can (or should) interact with that problem. That's a problem for architecture. > > So, you call closeFile immediately and read/write start > > returning null. > > And I have partially-read or partially-written data. I expect you flushed before killing the file. > Or Maybe I > call closeFile(), main thread continues and opens another file, > which gives the same file descriptor, `shared` references to > FileHandle which the user forgot to wait on continue to work > oblivious to the fact that it's a different file now. It's wild to suggest that ANY design for `shared` should somehow deal with the OS recycling a file handle... And it's still not an un-@safe crash! It's just a program with a bug. > It's a > horrible, but still @safe, implementation of FileHandle, yes, but > the caller (user) doesn't know that, and can't know that just > from the interface. The only advice against that is "don't do > that", but that's irrespective of your proposal. No proposal can (or should) address these issues. You're concerned with an issue that is so far left-field at this stage. Programming languages don't validate that you wrote a working bug-free program. > > I'm going to assume that `shareWithThreads()` was implemented > > by an > > 'expert' who checked the function results for errors. It was > > detected that the reads/write failed, and an error "failed to > > read file" was emit, then the function returned promptly. > > The uncertainty of what happens in this program is however > > `shareWithThreads()` handles read/write emitting an error. > > But you can only find out about these errors in `waitForThreads`, > the very call that the user "forgot" to make! ...what? > [ ... snip ... ] You have to concede defeat at this point. Destroy my proposal with another legitimately faulty program. > I understand that. So... it would seem that your proposal focuses > more on @safe than on threadsafety? I am trying to achieve @safe-ty _with respect to threadsafety_. 'threadsafety' with respect to "proper program operation" is a job for programmers, and program architecture. No language attribute can make your program right.
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 18:45:15 UTC, Walter Bright wrote: I'd like to add that if the compiler can prove that a T* points to a unique T, then it can be implicitly cast to shared(T)*. And it does so, like the result of .dup can be so converted. This can be achieved by using the unique struct and enforce the uniqueness at compile time. https://github.com/dlang/phobos/blob/master/std/typecons.d#L130
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 09:50:09 UTC, Walter Bright wrote:
Manu's Proposal ---
@safe:
int i;
int* a =
StartNewThread(a); // Compiles! Coder has no idea!
... in the new thread ...
void StartOfNewThread(shared(int)* b) {
... we have two threads accessing 'i',
one thinks it is shared, the other unshared,
and StartOfNewThread() has no idea and anyone
writing code for StartOfNewThread() has no way
to know anything is wrong ...
lockedIncrement(b); // Data Race!
No, does not compile, lockedIncrement takes an int*
Error cannot convert shared(int)* to int*
Your proposal means that the person writing the
lockedIncrement(), which is a perfectly reasonable thing to do,
Indeed.
simply cannot write it in a way that has a @safe interface,
because the person writing the lockedIncrement() library
function has no way to know that the data it receives is
actually unshared data.
It does, it takes an int* which is not implicitly convertible to
given an shared(int)*
I.e. @trusted code is obliged to proved a safe interface.
Yes.
Your proposal makes that impossible because the compiler would
allow unshared data to be implicitly typed as shared.
Yes, but not the other way around.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 12:00 PM Timon Gehr via Digitalmars-d wrote: > > On 21.10.18 17:54, Nicholas Wilson wrote: > > > >> As soon as that is done, you've got a data race with the other > >> existing unshared aliases. > > > > You're in @trusted code, that is the whole point. The onus is on the > > programmer to make that correct, same with regular @safe/@trusted@system > > code. > > Not all of the parties that participate in the data race are in @trusted > code. The point of @trusted is modularity: you manually check @trusted > code according to some set of restrictions and then you are sure that > there is no memory corruption. > > Note that you are not allowed to look at any of the @safe code while > checking your @trusted code. You will only see an opaque interface to > the @safe code that you call and all you know is that all the @safe code > type checks according to @safe rules. Note that there might be an > arbitrary number of @safe functions and methods that you do not see. > > Think about it this way: you first write all the @trusted and @system > code, and some evil guy who does not like you comes in after you looks > at your code and writes all the @safe code. If there is any memory > corruption, it will be your fault and you will face harsh consequences. > Now, design the @safe type checking rules. It won't be MP! > > Note that there may well be a good way to get the good properties of MP > without breaking the type system, but MP itself is not good because it > breaks @safe. Show me. Nobody has been able to show that yet. I'd really like to know this.
Re: shared - i need it to be useful
On 21.10.18 17:54, Nicholas Wilson wrote: As soon as that is done, you've got a data race with the other existing unshared aliases. You're in @trusted code, that is the whole point. The onus is on the programmer to make that correct, same with regular @safe/@trusted@system code. Not all of the parties that participate in the data race are in @trusted code. The point of @trusted is modularity: you manually check @trusted code according to some set of restrictions and then you are sure that there is no memory corruption. Note that you are not allowed to look at any of the @safe code while checking your @trusted code. You will only see an opaque interface to the @safe code that you call and all you know is that all the @safe code type checks according to @safe rules. Note that there might be an arbitrary number of @safe functions and methods that you do not see. Think about it this way: you first write all the @trusted and @system code, and some evil guy who does not like you comes in after you looks at your code and writes all the @safe code. If there is any memory corruption, it will be your fault and you will face harsh consequences. Now, design the @safe type checking rules. It won't be MP! Note that there may well be a good way to get the good properties of MP without breaking the type system, but MP itself is not good because it breaks @safe.
Re: shared - i need it to be useful
I'd like to add that if the compiler can prove that a T* points to a unique T, then it can be implicitly cast to shared(T)*. And it does so, like the result of .dup can be so converted.
Re: shared - i need it to be useful
On Sun, Oct 21, 2018 at 3:00 AM Walter Bright via Digitalmars-d wrote: > > On 10/20/2018 11:08 AM, Nicholas Wilson wrote: > > You can if no-one else writes to it, which is the whole point of Manu's > > proposal. Perhaps it should be const shared instead of shared but still. > > There is no purpose whatsoever to data that can be neither read nor written. There is no primitive type with implicit threadsafety... nor need there be. Shared data simply can not be read or written in any threadsafe manner. This is a rock-solid reality, and the type system needs to reflect that reality as a fundamental premise. Only from there can we start to define meaningful threadsafety. All threadsafe interactions with anything involve calling functions. It's completely reasonable to make `shared` inhibit all read and write access to data. We can only call shared methods, because only real-code can implement threadsafety. > Shared data is only useful if, at some point, it is read/written, presumably > by > casting it to unshared in @trusted code. As soon as that is done, you've got a > data race with the other existing unshared aliases. If such a race is possible, then the @trusted function is not threadsafe, so it is not @trusted by definition. You wrote a bad @trusted function, and you should feel bad. The simplest way to guarantee that no unsafe access is possible is to use encapsulation to assure no unregulated access exists.
Re: shared - i need it to be useful
On Sun., 21 Oct. 2018, 2:55 am Walter Bright via Digitalmars-d,
wrote:
>
> On 10/20/2018 11:24 AM, Manu wrote:
> > This is an unfair dismissal.
>
> It has nothing at all to do with fairness. It is about what the type system
> guarantees in @safe code. To repeat, the current type system guarantees in
> @safe
> code that T* and shared(T)* do not point to the same memory location.
>
> Does your proposal maintain that or not? It's a binary question.
By the definition Nick pulled from Wikipedia and posted for you a few
posts back, yes, my proposal satisfies Wikipedia's definition of no
aliasing. I understand that property is critical, and I have carefully
designed for it.
> > I'm not sure you've understood the proposal.
> > This is the reason for the implicit conversion. It provides safe
> > transition.
>
> I don't see any way to make an implicit T* to shared(T)* safe, or vice versa.
> The T* code can create more aliases that the conversion doesn't know about,
> and
> the shared(T)* code can hand out aliases to other threads. So it all falls to
> pieces.
T* can't make additional T* aliases on other threads; there can only
be one thread with T*.
shared(T)* can not make a T*.
shared(T)* has no read or write access, so it's not an alias of T* by
Wikipedia's definition.
Only threadsafe functions can do anything to T.
The leap of faith is; some @trusted utility functions at the bottom of
the shared stack makes a promise that it is threadsafe, and must
deliver that promise.
I don't think this is unreasonable; this is the nature of @trusted
functions, they make a promise, and they must keep it.
If the trusted function does not lie, then the chain of trust holds
upwards through the stack.
The are very few such trusted functions in practise. Like, similar to
the number of digits you have.
> Using a 'scope' qualifier won't work, because 'scope' isn't transitive,
> while shared is, i.e. U** and shared(U*)*.
I don't think I depend on scope in any way.
That was an earlier revision of thinking in an older thread.
> > I'm not sure how to clarify it, what can I give you?
>
> Write a piece of code that does such an implicit conversion that you argue is
> @safe. Make the code as small as possible. Your example:
>
> > int* a;
> > shared(int)* b = a;
>
> This is not safe.
>
> Manu's Proposal ---
> @safe:
> int i;
> int* a =
> StartNewThread(a); // Compiles! Coder has no idea!
>
> ... in the new thread ...
> void StartOfNewThread(shared(int)* b) {
>
> ... we have two threads accessing 'i',
> one thinks it is shared, the other unshared,
> and StartOfNewThread() has no idea and anyone
> writing code for StartOfNewThread() has no way
> to know anything is wrong ...
>
> lockedIncrement(b); // Data Race!
> }
This program doesn't compile. You receive an error because it is not safe.
The function is `lockedIncrement(int*)`. It can't receive a shared
argument; the function is not threadsafe by my definition.
You have written a program that produces the expected error that
alerts you that you have tried to do un-@safe and make a race.
Stanislav produced this same program, and I responded with the correct
program a few posts back.
I'll repeat it here; the @safe program to model this interaction is:
@safe:
// function is NOT threadsafe by my definition, can not be called on
shared arguments
void atomicIncrement(int*);
struct Atomic(T)
{
// encapsulare the unsafe data so it's inaccessible by any unsafe means
private T val;
// perform the unsafe cast in a trusted function
// we are able to assure a valid calling context by encapsulating
the data above
void opUnary(string op : "++")() shared @trusted {
atomicIncrement(cast(T*)); }
}
Atomic!int i;
Atomic!int* a =
StartNewThread(a); // Compiles, of course!
++i; // no race
... in the new thread ...
void StartOfNewThread(shared(Atomic!int)* b) {
... we have two threads accessing 'i', one has thread-local access,
this one has a restricted shared access.
here, we have a shared instance, so we can only access `b` via
threadsafe functions.
as such, we can manipulate `b` without fear.
++i; // no race!
}
> Your proposal means that the person writing the lockedIncrement(), which is a
> perfectly reasonable thing to do, simply cannot write it in a way that has a
> @safe interface
Correct, the rules of my proposal apply to lockedIncrement(). They
apply to `shared` generally.
lockedIncrement() is not a threadsafe function. You can't call it on a
shared instance, because `int`s API (ie, all intrinsic operations) are
not threadsafe.
lockedIncrement() can't promise threadsafe access to `shared(int)*`,
so the argument is not shared.
Your program made the correct compile error about doing unsafety, but
the location of the compile error is different under my proposal;
complexity is worn by the shared library author, rather than every
calling user ever.
I think my proposal places the complexity in the right location.
`shared` is
Re: shared - i need it to be useful
On Sat, 20 Oct 2018 22:47:14 -0700, Manu wrote: > Looking at the meat of the program; you open a file, and distribute it > to do accesses (I presume?) > Naturally, this is a really weird thing to do, because even if the API > is threadsafe such that it doesn't crash and reads/writes are > serialised, the sequencing of reads/writes will be random, so I don't > believe any sane person (let alone an expert) would write this > program... but moving on. > Then you wait for them to finish, and close the file. > > Fine. You have a file with randomly interleaved data... for whatever > reason. I'd expect almost every nontrivial multithreaded program to do this. It's called a log file. You don't need to read data pointed at by a log file, but you do need to read the FILE* or the file descriptor. Database-like things using a journal file might need a shared file for both reading and writing. > So, you call closeFile immediately and read/write start returning null. You start threads. You give them access to the log file. You wait for the threads to exit. Then you close the file.
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 09:58:18 UTC, Walter Bright wrote: On 10/20/2018 11:08 AM, Nicholas Wilson wrote: You can if no-one else writes to it, which is the whole point of Manu's proposal. Perhaps it should be const shared instead of shared but still. There is no purpose whatsoever to data that can be neither read nor written. Indeed but there is a subtle difference between that and Manu's proposal: access through the shared variable may not have non-atomic reads, not no reads. Shared data is only useful if, at some point, it is read/written, Yes presumably by casting it to unshared in @trusted code. That is one way to do it, others include atomics and other @trusted primitives As soon as that is done, you've got a data race with the other existing unshared aliases. You're in @trusted code, that is the whole point. The onus is on the programmer to make that correct, same with regular @safe/@trusted@system code.
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 13:24:49 UTC, Stanislav Blinov
wrote:
On Sunday, 21 October 2018 at 11:25:16 UTC, aliak wrote:
When I say ok, I mean assuming the implementer actually wrote
correct code. This applies to any shared method today as well.
This ("ok") can only be achieved if the "implementor" (the
"expert") writes every function self-contained, at which point
sharing something from user code becomes a non-issue (i.e. it
becomes unnecessary). But that's not a very useful API. As soon
as you have more than one function operating on the same data,
the onus is on the user (the caller) to call those functions in
correct order, or, more generally, without invalidating the
state of shared data.
The onus is *always* on the user to write function calls in the
correct order, multi-threading or not. We expect programmers to
be able to figure out why this doesn't print 'Hello world!':
void main() {
import std.stdio;
string hello;
writeln(hello);
hello = "Hello world!";
}
We also expect writeln to be written in such a way that it
doesn't corrupt random data or cause life-threatening situations
just because hello was uninitialized upon calling writeln, and
assigned afterwards. We should expect the same of multi-threaded
programs.
This places the onus of writing thread-safe code on the writer of
the multi-threaded equivalent of writeln and string.opAssign.
Only this way can the user of the library write code and not
expect things to blow up in their face.
--
Simen
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 12:45:43 UTC, Stanislav Blinov wrote: On Sunday, 21 October 2018 at 05:47:14 UTC, Manu wrote: On Sat, Oct 20, 2018 at 10:10 AM Stanislav Blinov via Digitalmars-d wrote: Synchronized with what? You still have `a`, which isn't `shared` and doesn't require any atomic access or synchronization. At this point it doesn't matter if it's an int or a struct. As soon as you share `a`, you can't just pretend that reading or writing `a` is safe. `b` can't read or write `a`... accessing `a` is absolutely safe. It's not, with or without your proposal. The purpose of sharing `a` into `b` is to allow someone to access `*a` in a threadsafe way (but un-@safe, as it *will* require casting away `shared` from `b`). That is what's making keeping an unshared reference `a` un-@safe: whoever accesses `*a` in their @trusted implementations via `*b` can't know that `*a` is being (@safe-ly!) accessed in a non-threadsafe way at the same time. Then someone has not done their job. Since the pieces of code that will actually use the un-@safe building blocks at the bottom are few and far between, it is reasonable to assume that an expert will be writing this code, and that such code be placed in a separate module where all access to the shared type is controlled. It seems you expect regular users to have calls to atomicOp!"++" scattered all over their code. I find this an unreasonable expectation, and fully agree that this will lead to problems. Someone must do something unsafe to undermine your threadsafety... and if you write unsafe code and don't know what you're doing, there's nothing that can help you. Ergo, it follows that anyone that is making an implicit cast from mutable to shared better know what they're doing, which mere mortal users (not "experts") might not. I.e. it's a way to giving a loaded gun to someone who never held a weapon before. No. Close does not promise threadsafety itself (but of course, it doesn't violate read/write's promise, or the program is invalid). Yep, and that's the issue. It SHALL NOT violate threadsafety, but it can't promise such in any way :( Can you demonstrate any system that can promise something like that? (apart from all-immutable) read and write will appropriately check their file-open state each time they perform their actions. Why? The only purpose of giving someone a `shared` reference is to give a reference to an open file. `shared` references can't do anything with the file but read and write, they would expect to be able to do so. Because otherwise it's not thread-safe. Exactly as you point out, the owner could call closeFile before some other thread was finished writing. If the implementer of FileHandle fails to take this into account, then no, it's not thread-safe. I'm going to assume that `shareWithThreads()` was implemented by an 'expert' who checked the function results for errors. It was detected that the reads/write failed, and an error "failed to read file" was emit, then the function returned promptly. The uncertainty of what happens in this program is however `shareWithThreads()` handles read/write emitting an error. But you can only find out about these errors in `waitForThreads`, the very call that the user "forgot" to make! Of course not. You can throw exceptions, you could add a destructor that reports on these errors, you could set an error flag somewhere and check that every now and then. The fact that you've managed to write a horribly broken API under MP and can't see a way to do better inside that system does not necessarily mean the problem is with MP. -- Simen
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 09:58:18 UTC, Walter Bright wrote:
On 10/20/2018 11:08 AM, Nicholas Wilson wrote:
You can if no-one else writes to it, which is the whole point
of Manu's proposal. Perhaps it should be const shared instead
of shared but still.
There is no purpose whatsoever to data that can be neither read
nor written. Shared data is only useful if, at some point, it
is read/written, presumably by casting it to unshared in
@trusted code. As soon as that is done, you've got a data race
with the other existing unshared aliases.
No, because every part of the public interface has to work
together to ensure thread-safety.
This code is invalid (but compiles) under MP:
module A;
struct S {
private int n;
void foo() @safe {
n--; // Not thread-safe
}
void bar() shared @trusted {
atomicOp!"++"(n.assumeUnshared);
}
}
module B;
import A;
void passToOtherThread(shared(S)*); // Calls S.bar()
void main() {
S* s = new S();
passToOtherThread(s);
s.foo();
}
The reason: foo() breaks bar()s promise of thread-safety. This
means that S does not provide a thread-safe interface. It would
be nice if the compiler could statically notice that, but I don't
see how that'd work.
Now, for a thread-safe version:
module A;
struct S {
int n;
void foo() @trusted {
atomicOp!"--"(n); // Thread-safe
}
void bar() shared @trusted {
atomicOp!"++"(n.assumeUnshared);
}
}
module B;
import A;
void passToOtherThread(shared(S)*); // Calls S.bar()
void main() {
S* s = new S();
passToOtherThread(s);
s.foo();
}
In this case, passToOtherThread is free to call S.bar as often as
it may feel like, since atomic operations are used in every
possible access to S.n. This is true even though one thread has
unshared access and other threads have shared access.
--
Simen
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 09:50:09 UTC, Walter Bright wrote:
On 10/20/2018 11:24 AM, Manu wrote:
This is an unfair dismissal.
It has nothing at all to do with fairness. It is about what the
type system guarantees in @safe code. To repeat, the current
type system guarantees in @safe code that T* and shared(T)* do
not point to the same memory location.
Does your proposal maintain that or not? It's a binary question.
No. Instead, it proposes something more useful: once cast to
shared(T)*, only thread-safe operations may be performed on it.
> int* a;
> shared(int)* b = a;
This is not safe.
Under MP, this is perfectly safe - you can do nothing with a
shared(int)*, except call un-@safe, non-thread-safe functions on
it, which will *fail to compile* under @safe.
Manu's Proposal ---
@safe:
int i;
int* a =
StartNewThread(a); // Compiles! Coder has no idea!
... in the new thread ...
void StartOfNewThread(shared(int)* b) {
... we have two threads accessing 'i',
one thinks it is shared, the other unshared,
and StartOfNewThread() has no idea and anyone
writing code for StartOfNewThread() has no way
to know anything is wrong ...
lockedIncrement(b); // Data Race!
}
Someone's messed up if they've marked lockedIncrement @safe -
under MP, it shouldn't be. lockedIncrement is a very low-level
piece of functionality, and should be @system. It also shouldn't
take a shared(int)*, but a int*, forcing an unsafe cast and
making it obvious the code is un@safe.
--
Simen
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 11:25:16 UTC, aliak wrote:
On Saturday, 20 October 2018 at 16:41:41 UTC, Stanislav Blinov
wrote:
Those are not "ok". They're only "ok" under Manu's proposal so
long as the author of C promises (via documentation) that
that's indeed "ok". There can be no statically-enforced
guarantees that those calls are "ok", or that issuing them in
that order is "ok". Yet Manu keeps insisting that somehow
there is.
No he is not insisting you can statically enforce thread safety.
I stand corrected, it would seem so.
When I say ok, I mean assuming the implementer actually wrote
correct code. This applies to any shared method today as well.
This ("ok") can only be achieved if the "implementor" (the
"expert") writes every function self-contained, at which point
sharing something from user code becomes a non-issue (i.e. it
becomes unnecessary). But that's not a very useful API. As soon
as you have more than one function operating on the same data,
the onus is on the user (the caller) to call those functions in
correct order, or, more generally, without invalidating the state
of shared data.
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 05:47:14 UTC, Manu wrote:
On Sat, Oct 20, 2018 at 10:10 AM Stanislav Blinov via
Digitalmars-d wrote:
Synchronized with what? You still have `a`, which isn't
`shared` and doesn't require any atomic access or
synchronization. At this point it doesn't matter if it's an
int or a struct. As soon as you share `a`, you can't just
pretend that reading or writing `a` is safe.
`b` can't read or write `a`... accessing `a` is absolutely safe.
It's not, with or without your proposal. The purpose of sharing
`a` into `b` is to allow someone to access `*a` in a threadsafe
way (but un-@safe, as it *will* require casting away `shared`
from `b`). That is what's making keeping an unshared reference
`a` un-@safe: whoever accesses `*a` in their @trusted
implementations via `*b` can't know that `*a` is being
(@safe-ly!) accessed in a non-threadsafe way at the same time.
Someone must do something unsafe to undermine your
threadsafety... and
if you write unsafe code and don't know what you're doing,
there's
nothing that can help you.
Ergo, it follows that anyone that is making an implicit cast from
mutable to shared better know what they're doing, which mere
mortal users (not "experts") might not. I.e. it's a way to giving
a loaded gun to someone who never held a weapon before.
Today, every interaction with shared is unsafe.
Nod.
Creating a safe interaction with shared will lead to people not
doing unsafe things at every step.
Triple nod.
Encapsulate it all you want, safety only remains a
contract of convention, the language can't enforce it.
You're talking about @trusted code again. You're fixated on
unsafe interactions... my proposal is about SAFE interactions.
I'm trying to obliterate unsafe interactions with shared.
I know... Manu, I *know* what you're trying to do. We (me, Atila,
Timon, Walter...) are not opposing your goals, we're pointing out
the weakest spot of your proposal, which, it would seem, would
require more changes to the language than just disallowing
reading/writing `shared` members.
module expertcode;
@safe:
struct FileHandle {
@safe:
void[] read(void[] storage) shared;
void[] write(const(void)[] buffer) shared;
}
FileHandle openFile(string path);
// only the owner can close
void closeFile(ref FileHandle);
void shareWithThreads(shared FileHandle*); // i.e. generate a
number of jobs in some queue
void waitForThreads(); // waits until all
processing is done
module usercode;
import expertcode;
void processHugeFile(string path) {
FileHandle file = openFile(path);
shareWithThreads();// implicit cast
waitForThreads();
file.closeFile();
}
This is a very strange program...
Why? That's literally the purpose of being able to `share`: you
create/acquire a resource, share it, but keep a non-`shared`
reference to yourself. If that's not required, you'd just create
the data `shared` to begin with.
I'm dubious it is in fact "expertcode"... but let's look into
it.
You're fixating on it being file now. I give an abstract example,
you dismiss it as contrived, I give a concrete one, you want to
dismiss it as "strange".
Heh, replace 'FileHandle' with 'BackBuffer', 'openFile' with
'acquireBackBuffer', 'shareWithThreads' with
'generateDrawCommands', 'waitForThreads' with
'gatherCommandsAndDraw', 'closeFile' with 'postProcessAndPresent'
;)
File handle seems to have just 2 methods... and they are both
threadsafe. Open and Close are free-functions.
It doesn't matter if they're free functions or not. What matters
is signature: they're taking non-`shared` (i.e. 'owned')
reference. Methods are free functions in disguise.
Close does not promise threadsafety itself (but of course, it
doesn't violate read/write's promise, or the program is
invalid).
Yep, and that's the issue. It SHALL NOT violate threadsafety, but
it can't promise such in any way :(
I expect the only possible way to achieve this is by an
internal mutex to make sure read/write/close calls are
serialised.
With that particular interface, yes.
read and write will appropriately check their file-open state
each time they perform their actions.
Why? The only purpose of giving someone a `shared` reference is
to give a reference to an open file. `shared` references can't do
anything with the file but read and write, they would expect to
be able to do so.
What read/write do in the case of being called on a closed
file... anyones guess? I'm gonna say they do no-op... they
return a null pointer to indicate the error state.
Looking at the meat of the program; you open a file, and
distribute it to do accesses (I presume?)
Naturally, this is a really weird thing to do, because even if
the API is threadsafe such that it doesn't crash and
reads/writes are
serialised, the sequencing of reads/writes will be random, so I
don't believe any sane person (let alone an expert) would write
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 16:41:41 UTC, Stanislav Blinov wrote: Those are not "ok". They're only "ok" under Manu's proposal so long as the author of C promises (via documentation) that that's indeed "ok". There can be no statically-enforced guarantees that those calls are "ok", or that issuing them in that order is "ok". Yet Manu keeps insisting that somehow there is. No he is not insisting you can statically enforce thread safety. When I say ok, I mean assuming the implementer actually wrote correct code. This applies to any shared method today as well.
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 09:58:18 UTC, Walter Bright wrote:
On 10/20/2018 11:08 AM, Nicholas Wilson wrote:
You can if no-one else writes to it, which is the whole point
of Manu's proposal. Perhaps it should be const shared instead
of shared but still.
There is no purpose whatsoever to data that can be neither read
nor written. Shared data is only useful if, at some point, it
is read/written, presumably by casting it to unshared in
@trusted code. As soon as that is done, you've got a data race
with the other existing unshared aliases.
Just a thought: if a hard requirement is made on `shared` data to
be non-copyable, a @safe conversion could be guaranteed. But it
can't be implicit either:
shared(T) share(T)(T value) if (!is(T == shared) &&
!isCopyable!T) {
shared(T) result = move(value);
return result;
}
struct ShareableData {
@disable ; // Generated by
compiler in presence of `shared` members and/or `shared` methods
/* ... */
}
void sendToThread(T)(shared T* ptr) @safe;
void usage() @safe {
int x;
sendToThread(); // Error: 'x' is not shared
shared y = x; // Ok
sendToThread(); // Ok
ShareableData data;
sendToThread(); // Error: 'data' is not shared
auto p =
sendToThread(p); // Error: *p is not shared
auto sharedData = share(move(data));
sendToThread(); // Ok
auto yCopy = y; // Error: cannot copy 'shared' y
auto dataCopy = sharedData; // Error: cannot copy 'shared'
sharedData
ShareableData otherData;
sendToThread(cast(shared(ShareableData)*) ); //
Error non-@safe cast in @safe code
}
And again, we're back to 'once it's shared, it can't be @safe-ly
unshared', which ruins the distinction between owned and shared
references, which is one of the nicer properties that Manu's
proposal seems to want to achieve :(
Re: shared - i need it to be useful
On 10/20/2018 11:08 AM, Nicholas Wilson wrote: You can if no-one else writes to it, which is the whole point of Manu's proposal. Perhaps it should be const shared instead of shared but still. There is no purpose whatsoever to data that can be neither read nor written. Shared data is only useful if, at some point, it is read/written, presumably by casting it to unshared in @trusted code. As soon as that is done, you've got a data race with the other existing unshared aliases.
Re: shared - i need it to be useful
On 21/10/2018 10:41 PM, Manu wrote: On Sun., 21 Oct. 2018, 2:05 am Walter Bright via Digitalmars-d, mailto:digitalmars-d@puremagic.com>> wrote: On 10/20/2018 11:30 AM, Manu wrote: > You can write an invalid program in any imaginable number of ways; > that's just not an interesting discussion. What we're discussing is not an invalid program, but what guarantees the type system can provide. D's current type system guarantees that a T* and a shared(T)* do not point to the same memory location in @safe code. My proposal guarantees that too, but in a more interesting way, because it opens the door to a whole working model. And it's totally @safe. To get them to point to the same memory location, you've got to dip into @system code, where *you* become responsible for maintaining the guarantees. My model preserves that property. Why do you think I'm running that static guarantee? It's all irrelevant if you don't express any mechanism to *do* anything. Shared today does not have any use. It simply expresses that data *is* shared, and says nothing about what you can do with it. If you don't express a safe mechanism for interacting with shared data, then simply expressing the distinction of shared data really is completely uninteresting. It's just a marker that's mixed up in a bunch of unsafe code. I'm no more satisfied than I am with C++. Shared needs to do something; I propose that it strictly models operations that are threadsafe and semantic restrictions required to support that, and then you have a *usage* scheme, which is safe, and API conveys proper interaction.. not just an uninteresting marker. I'm genuinely amazed that you're not intrigued by a @safe shared proposition. Nobly likes @safe more than you. I could run our entire SMP stack 100% @safe. I am going to fork D with this feature one way or another. It's the most meaningful and compelling opportunity I've seen in ever. If there's ever been a single thing that could truly move a bunch of C++ programmers, this is it. C++ can do a crappy job of modelling most stuff in D, but it simply can't go anywhere near this, and I've been working on competing C++ models for months. SMP is the future, we're going all-in this generation. Almost every function in our codebase runs in an SMP environment... And I was staggered that I was able to work this definition through to such a simple and elegant set of rules. I can't get my head around why people aren't more excited about this... fully @safe SMP is huge! I'm excited, but you need to write a DIP even if preliminary which shows both new semantics but also shows both working and current code to compare them.
Re: shared - i need it to be useful
On 10/20/2018 11:24 AM, Manu wrote:
This is an unfair dismissal.
It has nothing at all to do with fairness. It is about what the type system
guarantees in @safe code. To repeat, the current type system guarantees in @safe
code that T* and shared(T)* do not point to the same memory location.
Does your proposal maintain that or not? It's a binary question.
I'm not sure you've understood the proposal.
This is the reason for the implicit conversion. It provides safe
transition.
I don't see any way to make an implicit T* to shared(T)* safe, or vice versa.
The T* code can create more aliases that the conversion doesn't know about, and
the shared(T)* code can hand out aliases to other threads. So it all falls to
pieces. Using a 'scope' qualifier won't work, because 'scope' isn't transitive,
while shared is, i.e. U** and shared(U*)*.
> I'm not sure how to clarify it, what can I give you?
Write a piece of code that does such an implicit conversion that you argue is
@safe. Make the code as small as possible. Your example:
> int* a;
> shared(int)* b = a;
This is not safe.
Manu's Proposal ---
@safe:
int i;
int* a =
StartNewThread(a); // Compiles! Coder has no idea!
... in the new thread ...
void StartOfNewThread(shared(int)* b) {
... we have two threads accessing 'i',
one thinks it is shared, the other unshared,
and StartOfNewThread() has no idea and anyone
writing code for StartOfNewThread() has no way
to know anything is wrong ...
lockedIncrement(b); // Data Race!
}
--- Current D ---
@safe:
int i;
int* a =
StartNewThread(a); // Danger, Will Robinson! Does Not Compile!
StartNewThread(cast(shared(int)*) a) // Danger, Will Robinson!
// Unsafe Cast! Does Not Compile!
---
Your proposal means that the person writing the lockedIncrement(), which is a
perfectly reasonable thing to do, simply cannot write it in a way that has a
@safe interface, because the person writing the lockedIncrement() library
function has no way to know that the data it receives is actually unshared data.
I.e. @trusted code is obliged to proved a safe interface. Your proposal makes
that impossible because the compiler would allow unshared data to be implicitly
typed as shared.
Re: shared - i need it to be useful
On Sun., 21 Oct. 2018, 2:05 am Walter Bright via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: > On 10/20/2018 11:30 AM, Manu wrote: > > You can write an invalid program in any imaginable number of ways; > > that's just not an interesting discussion. > > What we're discussing is not an invalid program, but what guarantees the > type > system can provide. > > D's current type system guarantees that a T* and a shared(T)* do not point > to > the same memory location in @safe code. > My proposal guarantees that too, but in a more interesting way, because it opens the door to a whole working model. And it's totally @safe. To get them to point to the same memory location, you've got to dip into > @system > code, where *you* become responsible for maintaining the guarantees. > My model preserves that property. Why do you think I'm running that static guarantee? It's all irrelevant if you don't express any mechanism to *do* anything. Shared today does not have any use. It simply expresses that data *is* shared, and says nothing about what you can do with it. If you don't express a safe mechanism for interacting with shared data, then simply expressing the distinction of shared data really is completely uninteresting. It's just a marker that's mixed up in a bunch of unsafe code. I'm no more satisfied than I am with C++. Shared needs to do something; I propose that it strictly models operations that are threadsafe and semantic restrictions required to support that, and then you have a *usage* scheme, which is safe, and API conveys proper interaction.. not just an uninteresting marker. I'm genuinely amazed that you're not intrigued by a @safe shared proposition. Nobly likes @safe more than you. I could run our entire SMP stack 100% @safe. I am going to fork D with this feature one way or another. It's the most meaningful and compelling opportunity I've seen in ever. If there's ever been a single thing that could truly move a bunch of C++ programmers, this is it. C++ can do a crappy job of modelling most stuff in D, but it simply can't go anywhere near this, and I've been working on competing C++ models for months. SMP is the future, we're going all-in this generation. Almost every function in our codebase runs in an SMP environment... And I was staggered that I was able to work this definition through to such a simple and elegant set of rules. I can't get my head around why people aren't more excited about this... fully @safe SMP is huge! >
Re: shared - i need it to be useful
On Sunday, 21 October 2018 at 09:04:34 UTC, Walter Bright wrote: On 10/20/2018 11:30 AM, Manu wrote: You can write an invalid program in any imaginable number of ways; that's just not an interesting discussion. What we're discussing is not an invalid program, but what guarantees the type system can provide. D's current type system guarantees that a T* and a shared(T)* do not point to the same memory location in @safe code. To get them to point to the same memory location, you've got to dip into @system code, where *you* become responsible for maintaining the guarantees. The only difference between this and Manu's proposal is when you need to dip into @system code - in MP it's perfectly fine for the pointers to be equal, but when you want to read from or write to the address, you'll need to use @system. In other words, the dip into @system happens deeper in the codebase, meaning more code can be @safe. -- Simen
Re: shared - i need it to be useful
On 10/20/2018 11:30 AM, Manu wrote: You can write an invalid program in any imaginable number of ways; that's just not an interesting discussion. What we're discussing is not an invalid program, but what guarantees the type system can provide. D's current type system guarantees that a T* and a shared(T)* do not point to the same memory location in @safe code. To get them to point to the same memory location, you've got to dip into @system code, where *you* become responsible for maintaining the guarantees.
Re: shared - i need it to be useful
On Sat, Oct 20, 2018 at 10:10 AM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Saturday, 20 October 2018 at 16:48:05 UTC, Nicholas Wilson
> wrote:
> > On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright
> > wrote:
> >> On 10/19/2018 11:18 PM, Manu wrote:
> >>> The reason I ask is because, by my definition, if you have:
> >>> int* a;
> >>> shared(int)* b = a;
> >>>
> >>> While you have 2 numbers that address the same data, it is
> >>> not actually aliased because only `a` can access it.
> >>
> >> They are aliased,
> >
> > Quoting Wikipedia:
> >
> >>two pointers A and B which have the same value, then the name
> >>A[0] aliases the name B[0]. In this case we say the pointers A
> >>and B alias each other. Note that the concept of pointer
> >>aliasing is not very well-defined – two pointers A and B may or
> >>may not alias each other, depending on what operations are
> >>performed in the function using A and B.
> >
> > In this case given the above: `a[0]` does not alias `b[0]`
> > because `b[0]` is ill defined under Manu's proposal, because
> > the memory referenced by `a` is not reachable through `b`
> > because you can't read or write through `b`.
> >
> >> by code that believes it is unshared
> >
> > you cannot `@safe`ly modify the memory through `b`, `a`'s view
> > of the memory is unchanged in @safe code.
>
> And that's already a bug, because the language can't enforce
> threadsafe access through `a`, regardless of presence of `b`.
> Only the programmer can.
>
> >> and, code that believes it is shared.
> >
> > you cannot have non-atomic access though `b`, `b` has no @safe
> > view of the memory, unless it is atomic (which by definition is
> > synchronised).
>
> Synchronized with what? You still have `a`, which isn't `shared`
> and doesn't require any atomic access or synchronization. At this
> point it doesn't matter if it's an int or a struct. As soon as
> you share `a`, you can't just pretend that reading or writing `a`
> is safe.
`b` can't read or write `a`... accessing `a` is absolutely safe.
Someone must do something unsafe to undermine your threadsafety... and
if you write unsafe code and don't know what you're doing, there's
nothing that can help you.
Today, every interaction with shared is unsafe. Creating a safe
interaction with shared will lead to people not doing unsafe things at
every step.
> Encapsulate it all you want, safety only remains a
> contract of convention, the language can't enforce it.
You're talking about @trusted code again. You're fixated on unsafe
interactions... my proposal is about SAFE interactions. I'm trying to
obliterate unsafe interactions with shared.
> module expertcode;
>
> @safe:
>
> struct FileHandle {
> @safe:
>
> void[] read(void[] storage) shared;
> void[] write(const(void)[] buffer) shared;
> }
>
> FileHandle openFile(string path);
> // only the owner can close
> void closeFile(ref FileHandle);
>
> void shareWithThreads(shared FileHandle*); // i.e. generate a
> number of jobs in some queue
> void waitForThreads(); // waits until all
> processing is done
>
> module usercode;
>
> import expertcode;
>
> void processHugeFile(string path) {
> FileHandle file = openFile(path);
> shareWithThreads();// implicit cast
> waitForThreads();
> file.closeFile();
> }
This is a very strange program... I'm dubious it is in fact
"expertcode"... but let's look into it.
File handle seems to have just 2 methods... and they are both threadsafe.
Open and Close are free-functions. Close does not promise threadsafety
itself (but of course, it doesn't violate read/write's promise, or the
program is invalid).
I expect the only possible way to achieve this is by an internal mutex
to make sure read/write/close calls are serialised. read and write
will appropriately check their file-open state each time they perform
their actions. What read/write do in the case of being called on a
closed file... anyones guess? I'm gonna say they do no-op... they
return a null pointer to indicate the error state.
Looking at the meat of the program; you open a file, and distribute it
to do accesses (I presume?)
Naturally, this is a really weird thing to do, because even if the API
is threadsafe such that it doesn't crash and reads/writes are
serialised, the sequencing of reads/writes will be random, so I don't
believe any sane person (let alone an expert) would write this
program... but moving on.
Then you wait for them to finish, and close the file.
Fine. You have a file with randomly interleaved data... for whatever reason.
> Per your proposal, everything in 'expertcode' can be written
> @safe, i.e. not violating any of the points that @safe forbids,
> or doing so only in a @trusted manner. As far as the language is
> concerned, this would mean that processHugeFile can be @safe as
> well.
This program does appear to be safe (assuming that the implementations
aren't invalid), but a very strange program nonetheless.
>
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 18:30:59 UTC, Manu wrote:
On Sat, Oct 20, 2018 at 9:45 AM Stanislav Blinov via
Digitalmars-d wrote:
On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:
> class C {
> void f();
> void g() shared;
> }
Those are not "ok". They're only "ok" under Manu's proposal so
long as the author of C promises (via documentation) that
that's indeed "ok". There can be no statically-enforced
guarantees that those calls are "ok", or that issuing them in
that order is "ok". Yet Manu keeps insisting that somehow
there is.
I only insist that if you write a shared method, you promise
that it is threadsafe.
If f() undermines g() threadsafety, then **g() is NOT
threadsafe**, and you just write an invalid program.
You can write an invalid program in any imaginable number of
ways; that's just not an interesting discussion. An interesting
discussion is what we might to to help prevent writing such an
invalid program... I don't suggest here what we can do to
statically encorce this, but I suspect there does exist *some*
options which may help, which can be further developments.
What I also assert is that *this unsafe code is rare*... it
exists only at the bottom of the tooling stack, and anyone else
using a shared object will not do unsafe, and therefore will
not be able to create the problem. If you do unsafety anywhere
near `shared`, you should feel nervous. I'm trying to make a
world where you aren't *required* to do unsafety at every
single interaction.
Understand: f() can only undermine g() promise of threadsafety
**if f() is not @safe**. Users won't create this situation
accidentally, they can only do it deliberately.
---
module expertcode;
@safe:
struct FileHandle {
@safe:
void[] read(void[] storage) shared;
void[] write(const(void)[] buffer) shared;
}
FileHandle openFile(string path);
// only the owner can close
void closeFile(ref FileHandle);
void shareWithThreads(shared FileHandle*); // i.e. generate a
number of jobs in some queue
void waitForThreads(); // waits until all
processing is done
module usercode;
import expertcode;
void processHugeFile(string path) {
FileHandle file = openFile(path);
shareWithThreads();// implicit cast
waitForThreads();
file.closeFile();
}
---
Per your proposal, everything in 'expertcode' can be written
@safe, i.e. not violating any of the points that @safe forbids,
or doing so only in a @trusted manner. As far as the language is
concerned, this would mean that processHugeFile can be @safe as
well.
Remove the call to `waitForThreads()` (assume user just forgot
that, i.e. the "accident"). Nothing would change for the
compiler: all calls remain @safe. And yet, if we're lucky, we get
a consistent instacrash. If we're unlucky, we get memory
corruption, or an unsolicited write to another currently open
file, either of which can go unnoticed for some time.
Of course the program becomes invalid if you do that, there's no
question about it, this goes for all buggy code. The problem is,
definition of "valid" lies beyond the type system: it's an
agreement between different parts of code, i.e. between expert
programmers who wrote FileHandle et al., and users who write
processHugeFile(). The main issue is that certain *runtime*
conditions can still violate @safe-ty.
Your proposal makes the language more strict wrt. to writing
@safe 'expertmodule', thanks to disallowing reads and writes
through `shared`, which is great.
However the implicit conversion to `shared` doesn't in any way
improve the situation as far as user code is concerned, unless
I'm still missing something.
Re: shared - i need it to be useful
On Sat, Oct 20, 2018 at 9:45 AM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:
>
> > class C {
> > void f();
> > void g() shared;
> > }
> >
> > void t1(shared C c) {
> > c.g; // ok
> > c.f; // error
> > }
> >
> > void t2(shared C c) {
> > c.g; // ok
> > c.f; // error
> > }
> >
> > auto c = new C();
> > spawn(, c);
> > spawn(, c);
> > c.f; // ok
> > c.g; // ok
>
> Those are not "ok". They're only "ok" under Manu's proposal so
> long as the author of C promises (via documentation) that that's
> indeed "ok". There can be no statically-enforced guarantees that
> those calls are "ok", or that issuing them in that order is "ok".
> Yet Manu keeps insisting that somehow there is.
I only insist that if you write a shared method, you promise that it
is threadsafe.
If f() undermines g() threadsafety, then **g() is NOT threadsafe**,
and you just write an invalid program.
You can write an invalid program in any imaginable number of ways;
that's just not an interesting discussion. An interesting discussion
is what we might to to help prevent writing such an invalid program...
I don't suggest here what we can do to statically encorce this, but I
suspect there does exist *some* options which may help, which can be
further developments.
What I also assert is that *this unsafe code is rare*... it exists
only at the bottom of the tooling stack, and anyone else using a
shared object will not do unsafe, and therefore will not be able to
create the problem. If you do unsafety anywhere near `shared`, you
should feel nervous. I'm trying to make a world where you aren't
*required* to do unsafety at every single interaction.
Understand: f() can only undermine g() promise of threadsafety **if
f() is not @safe**. Users won't create this situation accidentally,
they can only do it deliberately.
Re: shared - i need it to be useful
On Sat, Oct 20, 2018 at 2:05 AM Walter Bright via Digitalmars-d wrote: > > On 10/19/2018 11:18 PM, Manu wrote: > > The reason I ask is because, by my definition, if you have: > > int* a; > > shared(int)* b = a; > > > > While you have 2 numbers that address the same data, it is not actually > > aliased > > because only `a` can access it. > > They are aliased, by code that believes it is unshared, and code that believes > it is shared. This situation could only occur if you do unsafe code badly. Unlike today, where you must do unsafe code to do any interaction of any kind, you will be able to do fully-safe interaction with the stack of tooling. In that world, any unsafe code and *particularly* where it interacts with shared will be an obnoxious and scary code smell. If it was possible to interact with shared safely, it would be blindingly suspicious when people are likely to be shooting themselves in the foot. The situation you describe here is *exactly* what we have right now, and I'm trying to prevent that. > This is not going to work This is an unfair dismissal. Have you tried it? I have. Write me the rules in a patch that I can take for a drive and demonstrate what the stack looks like. > > Exclusively distinguishing shared and unshared data is not an interesting > > distinction if shared data has no access. > > Somehow, you still have to find a way to give the shared path access, through > a > gate or a cast or a lock or whatever. I'm not sure you've understood the proposal. This is the reason for the implicit conversion. It provides safe transition. That's why I'm so insistent on it. > And then it breaks, because two different > threads are accessing the same data each thinking that data is not shared. This can only occur if you deliberately violate your @safety, and then mess up. This is *exactly* the interaction prescribed to shared today. This is what I'm fixing by making a fully @safe path! I think you demonstrate here that you haven't understood the reason, or the semantics of my proposal. I'm not sure how to clarify it, what can I give you?
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 17:06:22 UTC, Stanislav Blinov wrote: On Saturday, 20 October 2018 at 16:48:05 UTC, Nicholas Wilson wrote: On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright wrote: by code that believes it is unshared you cannot `@safe`ly modify the memory through `b`, `a`'s view of the memory is unchanged in @safe code. And that's already a bug, because the language can't enforce threadsafe access through `a`, regardless of presence of `b`. Only the programmer can. access through `a` is through the owned reference threadsafety through a does't mean anything, all _other_ access must ensure that the are ordered correctly. and, code that believes it is shared. you cannot have non-atomic access though `b`, `b` has no @safe view of the memory, unless it is atomic (which by definition is synchronised). Synchronized with what? You still have `a`, which isn't `shared` and doesn't require any atomic access or synchronization. Synchronized w.r.t any writes to that memory, e.g. from `a`. At this point it doesn't matter if it's an int or a struct. Yes. As soon as you share `a`, you can't just pretend that reading or writing `a` is safe. You can if no-one else writes to it, which is the whole point of Manu's proposal. Perhaps it should be const shared instead of shared but still.
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 16:41:41 UTC, Stanislav Blinov
wrote:
On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:
class C {
void f();
void g() shared;
}
void t1(shared C c) {
c.g; // ok
c.f; // error
}
void t2(shared C c) {
c.g; // ok
c.f; // error
}
auto c = new C();
spawn(, c); // line 20
spawn(, c); // line 21
c.f; // ok
c.g; // ok // line 23
Those are not "ok". They're only "ok" under Manu's proposal so
long as the author of C promises (via documentation) that
that's indeed "ok". There can be no statically-enforced
guarantees that those calls are "ok", or that issuing them in
that order is "ok". Yet Manu keeps insisting that somehow there
is.
Backing up a bit and making a few observations (after adding
imports and wrapping the bottom code in a function):
1. the code above currently does not compile, error messages are:
i. line 20 & 21: spawn fails to instantiate because c is not
shared
ii. line 23: shared method C.g is not callable using a
non-shared object
iii. the lines already marked // error
2. in order to fix 1.i, one must cast c to shared at the call
site, this is not @safe
3 fixing 1.ii requires doing `(cast(shared)c).g`, this is also
not @safe
4 fixing 1.iii fixing requires casting away shared, this is not
only not @safe, but also wrong. c is a class so one could try
locking it although I'm not sure what the implications are for
doing that when another thread owns the data, probably bad.
5 the current means of dealing with shared with lock and cast
away shared is also not @safe
6 under Manu's proposal reading and writing shared objects
results in compilation error
7 The static guarantees we have in the language are type safety
and @safe
8 under Manu's proposal to do anything one must call shared
functions on said object, this implies a "@trusted"
implementation at the bottom of the stack for ensuring thread
safety (atomics and lock + cast (assuming it is not wrong), other
sync primitives) that are not @safe, but not outright wrong
either.
The question then becomes: assuming the implementation _is_ @safe
type correct and thread safe etc., can the author of C provide
guarantees of @safe and type correctness? and can this guarantee
be free of false positives?
Currently the answer is no: the requirement to cast to and from
shared is un-@safe and that burden is on the user which means
that they must understand the inner workings of C to know it that
is the case.
Manu's proposal is slightly more interesting. shared becomes a
guarantee that accesses to that object will not race, assuming
that the @trusted implementation at the bottom of the stack are
correct. In the above if t1 and t2 took `const shared C` and `g`
was also const shared, then I think that it could.
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 16:48:05 UTC, Nicholas Wilson wrote: On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright wrote: On 10/19/2018 11:18 PM, Manu wrote: The reason I ask is because, by my definition, if you have: int* a; shared(int)* b = a; While you have 2 numbers that address the same data, it is not actually aliased because only `a` can access it. They are aliased, Quoting Wikipedia: two pointers A and B which have the same value, then the name A[0] aliases the name B[0]. In this case we say the pointers A and B alias each other. Note that the concept of pointer aliasing is not very well-defined – two pointers A and B may or may not alias each other, depending on what operations are performed in the function using A and B. In this case given the above: `a[0]` does not alias `b[0]` because `b[0]` is ill defined under Manu's proposal, because the memory referenced by `a` is not reachable through `b` because you can't read or write through `b`. by code that believes it is unshared you cannot `@safe`ly modify the memory through `b`, `a`'s view of the memory is unchanged in @safe code. And that's already a bug, because the language can't enforce threadsafe access through `a`, regardless of presence of `b`. Only the programmer can. and, code that believes it is shared. you cannot have non-atomic access though `b`, `b` has no @safe view of the memory, unless it is atomic (which by definition is synchronised). Synchronized with what? You still have `a`, which isn't `shared` and doesn't require any atomic access or synchronization. At this point it doesn't matter if it's an int or a struct. As soon as you share `a`, you can't just pretend that reading or writing `a` is safe. Encapsulate it all you want, safety only remains a contract of convention, the language can't enforce it.
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright wrote: On 10/19/2018 11:18 PM, Manu wrote: The reason I ask is because, by my definition, if you have: int* a; shared(int)* b = a; While you have 2 numbers that address the same data, it is not actually aliased because only `a` can access it. They are aliased, Quoting Wikipedia: two pointers A and B which have the same value, then the name A[0] aliases the name B[0]. In this case we say the pointers A and B alias each other. Note that the concept of pointer aliasing is not very well-defined – two pointers A and B may or may not alias each other, depending on what operations are performed in the function using A and B. In this case given the above: `a[0]` does not alias `b[0]` because `b[0]` is ill defined under Manu's proposal, because the memory referenced by `a` is not reachable through `b` because you can't read or write through `b`. by code that believes it is unshared you cannot `@safe`ly modify the memory through `b`, `a`'s view of the memory is unchanged in @safe code. and, code that believes it is shared. you cannot have non-atomic access though `b`, `b` has no @safe view of the memory, unless it is atomic (which by definition is synchronised). This is not going to work. Aú contraire.
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 16:18:53 UTC, aliak wrote:
class C {
void f();
void g() shared;
}
void t1(shared C c) {
c.g; // ok
c.f; // error
}
void t2(shared C c) {
c.g; // ok
c.f; // error
}
auto c = new C();
spawn(, c);
spawn(, c);
c.f; // ok
c.g; // ok
Those are not "ok". They're only "ok" under Manu's proposal so
long as the author of C promises (via documentation) that that's
indeed "ok". There can be no statically-enforced guarantees that
those calls are "ok", or that issuing them in that order is "ok".
Yet Manu keeps insisting that somehow there is.
Re: shared - i need it to be useful
On Saturday, 20 October 2018 at 09:04:17 UTC, Walter Bright wrote:
Somehow, you still have to find a way to give the shared path
access, through a gate or a cast or a lock or whatever. And
then it breaks, because two different threads are accessing the
same data each thinking that data is not shared.
When you say that, then under Manu's proposal and the code below:
class C {
void f();
void g() shared;
}
void t1(shared C c) {
c.g; // ok
c.f; // error
}
void t2(shared C c) {
c.g; // ok
c.f; // error
}
auto c = new C();
spawn(, c);
spawn(, c);
c.f; // ok
c.g; // ok
Do you mean the implementation of C.g? Since that is shared
wouldn't that just be a normal understanding that you'd need to
synchronize the data access in shared since it's a shared method?
And if you mean C.f, then if that accessed data (that was
accessed by C.g) unsafely, then that's just a bad implementation
no? Or?
Cheers,
- Ali
Re: shared - i need it to be useful
On 10/19/2018 11:18 PM, Manu wrote: The reason I ask is because, by my definition, if you have: int* a; shared(int)* b = a; While you have 2 numbers that address the same data, it is not actually aliased because only `a` can access it. They are aliased, by code that believes it is unshared, and code that believes it is shared. This is not going to work. Exclusively distinguishing shared and unshared data is not an interesting distinction if shared data has no access. Somehow, you still have to find a way to give the shared path access, through a gate or a cast or a lock or whatever. And then it breaks, because two different threads are accessing the same data each thinking that data is not shared.
Re: shared - i need it to be useful
On Fri, Oct 19, 2018 at 9:45 AM Steven Schveighoffer via Digitalmars-d
wrote:
>
> On 10/18/18 9:09 PM, Manu wrote:
> > On Thu, Oct 18, 2018 at 5:30 PM Timon Gehr via Digitalmars-d
> > wrote:
> >>
> >> On 18.10.18 23:34, Erik van Velzen wrote:
> >>> If you have an object which can be used in both a thread-safe and a
> >>> thread-unsafe way that's a bug or code smell.
> >>
> >> Then why do you not just make all members shared? Because with Manu's
> >> proposal, as soon as you have a shared method, all members effectively
> >> become shared.
> >
> > No they don't, only facets that overlap with the shared method.
> > I tried to present an example before:
> >
> > struct Threadsafe
> > {
> >int x;
> >Atomic!int y;
> >void foo() shared { ++y; } // <- shared interaction only affects 'y'
> >void bar() { ++x; ++y; } // <- not threadsafe, but does not violate
> > foo's commitment; only interaction with 'y' has any commitment
> > associated with it
> >void unrelated() { ++x; } // <- no responsibilities are transposed
> > here, you can continue to do whatever you like throughout the class
> > where 'y' is not concerned
> > }
> >
> > In practise, and in my direct experience, classes tend to have exactly
> > one 'y', and either zero (pure utility), or many such 'x' members.
> > Threadsafe API interacts with 'y', and the rest is just normal
> > thread-local methods which interact with all members thread-locally,
> > and may also interact with 'y' while not violating any threadsafety
> > commitments.
>
> I promised I wouldn't respond, I'm going to break that (obviously).
>
> But that's because after reading this description I ACTUALLY understand
> what you are looking for.
>
> I'm going to write a fuller post later, but I can't right now. But the
> critical thing here is, you want a system where you can divvy up a type
> into pieces you share and pieces you don't. But then you *don't* want to
> have to share only the shared pieces. You want to share the whole thing
> and be sure that it can't access your unshared pieces.
>
> This critical requirement makes things a bit more interesting. For the
> record, the most difficult thing to reaching this understanding was that
> whenever I proposed anything, your answer was something like 'I just
> can't work with that', and when I asked why, you said 'because it's
> useless', etc. Fully explaining this point is very key to understanding
> your thinking.
>
> To be continued...
I'm glad that there's movement here... but I'm still not 100%
convinced you understood me; perhaps getting close though.
I only say that because your description above has a whole lot more
words and complexity than is required to express my proposal. If you
perceive that complexity in structural terms, then I am still not
clearly understood.
> "divvy up a type into pieces"
This is an odd mental model of what I'm saying, and I can't sympathise
with those words, but if they work for you, and we agree on the
semantics, then sure...
If you write an object with some const methods and some non-const
methods, then take a const instance of the object... you can only call
the const methods. Have you 'divvied up the type' into a const portion
and a non-const portion? If the answer is yes, then I can accept your
description.
I would talk in terms of restriction:
An object has 4 functions, 2 are mutable, 2 are const... you apply
const to the type and you are *restricted* to only calling the 2 const
functions.
An object has 4 functions, 2 are unsahred, 2 are shared... you apply
shared to the type and you are *restricted* to only calling the 2
shared (threadsafe) functions.
I haven't 'broken the type up', I'm just restricting what you can do
to it from within a particular context. In the const context, you
can't mutate it. In the shared context, you can't do un-threadsafe to
it, and the guarantee of that is embedded in the rules:
1. shared data can not be read or written
2. shared methods must be threadsafe
a. this may require that they be @trusted at the low-level, like the
methods of `Atomic(T)`
b. no other method may violate the shared method's promise,
otherwise it does not actually deliver its promise
i. I have extensive experience with this and it's just not a
problem in practise, but compiler technology to assist would be
welcome!
ii. This does NOT mean the not-shared methods are somehow shared;
they just need to be careful when interacting with some (usually
small) subset of members
This design implies strong encapsulation, but that's a naturally
occurring tendency implementing anything that's threadsafe.
As a helpful best-practise to assure that non-shared methods don't
undermine a shared method's commitment; prefer to interact with
volatile members via accessors or properties that are themselves
shared (can be private if you like).
You will find that it's not actually hard to deliver on the object's commitment.
If you write tooling that is at the level one-up from the
Re: shared - i need it to be useful
On Fri., 19 Oct. 2018, 3:10 am Walter Bright via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: > On 10/17/2018 12:20 AM, Manu wrote: > > What does it mean 'aliased' precisely? > > Aliasing means there are two paths to the same piece of data. That could > be two > pointers pointing to the same data, or one pointer to a variable that is > accessible by name. > The reason I ask is because, by my definition, if you have: int* a; shared(int)* b = a; While you have 2 numbers that address the same data, it is not actually aliased because only `a` can access it. It is not aliased in any practical sense. > It doesn't really give us > > anything in practice that we don't have in C++. > > It provides a standard, enforced way to distinguish shared data from > unshared > data, and no way to bypass it in @safe code. There's no way to do that in > C++. > Right, but we can do so much better. I want shared to model "what is thread-@safe to do", because that models what you are able to do, and what API's should encourage when operating on `shared` things. Exclusively distinguishing shared and unshared data is not an interesting distinction if shared data has no access. I've been trying to say over and over; ignore what you think you know about that definition, accept my rules strictly as given (they're very simple and concise, there's only 2 rules), such that shared will mean "is threadsafe to call with this data" when applied to function args... Build the thought experiment outward from there. That's an interesting and useful definition for shared, and it leads to a framework where shared is useful in a fully @safe SMP program, and even models @safe transitions across unshared -> shared boundaries (parallel for, map/reduce, etc, fork and join style workloads), which are typical lock-free patterns. Lock-and-cast semantics are preserved unchanged for those that interact in the way shared is prescribed today, but strictly modeling that workflow is uninteresting, because it's unsafe by definition. I'm not losing that, but I'm trying to introduce a safe workflow that exists in complement, and my model works. I don't even know if we have a mutex defined in our codebase, we don't use them... but we can max out a 64core thread ripper. >
Re: shared - i need it to be useful
On Friday, 19 October 2018 at 18:00:47 UTC, Atila Neves wrote: Because int or int* does not have threadsafe member functions. https://dlang.org/phobos/core_atomic.html Atomic and thread-safe are two very different concepts. Thread-safe is more of an ecosystem thing - if there are ways to do non-thread-safe operations on something, atomic operations are not enough to make things thread-safe. This is what core.atomic does: https://i.imgur.com/PnKMigl.jpg If you close the other openings, atomics are fantastic building blocks to making something thread-safe, but on their own they are not enough. -- Simen
Re: shared - i need it to be useful
On 20/10/2018 2:07 AM, Dominikus Dittes Scherkl wrote: This document provide no reasoning about what usecases it supports: It was a basic idea of mine... It was never meant to be PR'd.
Re: shared - i need it to be useful
On Thursday, 18 October 2018 at 21:24:53 UTC, jmh530 wrote: On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote: [snip] Assuming this world... how do you use shared? https://github.com/atilaneves/fearless I had posted your library before to no response... I had two questions, if you'll indulge me. The first is perhaps more wrt automem. I noticed that I couldn't use automem's Unique with @safe currently. Is there any way to make it @safe, perhaps with dip1000? Yeah, I punted on making anything there @safe. I have to go back and fix it. At least I wrote Vector from scratch to be @safe. Second, Rust's borrow checker is that you can only have one mutable borrow. This is kind of like Exclusive, but it does it at compile-time, rather than with GC/RC. Is this something that can be incorporated into fearless? Well, Rust's version of Exclusive is Mutex, and that's pretty much always used with Arc. The closest we have to a borrow checker is DIP1000 and that's what fearless relies on.
Re: shared - i need it to be useful
On Thursday, 18 October 2018 at 19:04:58 UTC, Erik van Velzen wrote: On Thursday, 18 October 2018 at 17:47:29 UTC, Stanislav Blinov wrote: On Thursday, 18 October 2018 at 17:17:37 UTC, Atila Neves wrote: On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? int i; tid.send(); ++i; // oops, data race Doesn't work. No matter what you show Manu or Simen here they think it's just a bad contrived example. You can't sway them by the fact that the compiler currently *prevents* this from happening. Manu said clearly that the receiving thread won't be able to read or write the pointer. Not directly - but obviously there must be *some* way to using it, in this case since it's an int with one of the core.atomic functions. At that point the spawned thread will be accessing it correctly, but the parent thread can modify the int in a non-atomic fashion. Because int or int* does not have threadsafe member functions. https://dlang.org/phobos/core_atomic.html
Re: shared - i need it to be useful
On Fri., 19 Oct. 2018, 6:10 am Dominikus Dittes Scherkl via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: > On Friday, 19 October 2018 at 06:25:00 UTC, rikki cattermole > wrote: > > On 19/10/2018 7:09 PM, Norm wrote: > > > [0] > > https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md > > This document provide no reasoning about what usecases it > supports: > > Is it possible to create objects that are shared just for short > periods during their livetime and guarantee that they can be used > threadsave like Manu want it to be? > > Does it prohibit misuse any better than Manus proposal (that > requires the "Expert" to implement all theadsave API)? > No, a key misunderstanding. My proposal is @safe. The only thing an expert must do is write the few @trusted implementations that live at the very bottom of the stack. That would always be in a lib. When was the last time you rewrote std::map because you thought you could do better? The whole stack from there on up (the user stack) is safe, and you can have confidence in the @safe-ty. My goal is to make it safe, clearly communicate how a user interact with the API, and mechanically confirm that users do the right stuff. My proposal is specifically structured to not require *any* unsafe interactions at the user level. Only core machinery that is @trusted needs expert attention. I don't think it's possible to invent a proposal with a higher degree of verifiable safety.
Re: shared - i need it to be useful
On 10/18/18 9:09 PM, Manu wrote:
On Thu, Oct 18, 2018 at 5:30 PM Timon Gehr via Digitalmars-d
wrote:
On 18.10.18 23:34, Erik van Velzen wrote:
If you have an object which can be used in both a thread-safe and a
thread-unsafe way that's a bug or code smell.
Then why do you not just make all members shared? Because with Manu's
proposal, as soon as you have a shared method, all members effectively
become shared.
No they don't, only facets that overlap with the shared method.
I tried to present an example before:
struct Threadsafe
{
int x;
Atomic!int y;
void foo() shared { ++y; } // <- shared interaction only affects 'y'
void bar() { ++x; ++y; } // <- not threadsafe, but does not violate
foo's commitment; only interaction with 'y' has any commitment
associated with it
void unrelated() { ++x; } // <- no responsibilities are transposed
here, you can continue to do whatever you like throughout the class
where 'y' is not concerned
}
In practise, and in my direct experience, classes tend to have exactly
one 'y', and either zero (pure utility), or many such 'x' members.
Threadsafe API interacts with 'y', and the rest is just normal
thread-local methods which interact with all members thread-locally,
and may also interact with 'y' while not violating any threadsafety
commitments.
I promised I wouldn't respond, I'm going to break that (obviously).
But that's because after reading this description I ACTUALLY understand
what you are looking for.
I'm going to write a fuller post later, but I can't right now. But the
critical thing here is, you want a system where you can divvy up a type
into pieces you share and pieces you don't. But then you *don't* want to
have to share only the shared pieces. You want to share the whole thing
and be sure that it can't access your unshared pieces.
This critical requirement makes things a bit more interesting. For the
record, the most difficult thing to reaching this understanding was that
whenever I proposed anything, your answer was something like 'I just
can't work with that', and when I asked why, you said 'because it's
useless', etc. Fully explaining this point is very key to understanding
your thinking.
To be continued...
-Steve
Re: shared - i need it to be useful
On Friday, 19 October 2018 at 06:25:00 UTC, rikki cattermole wrote: On 19/10/2018 7:09 PM, Norm wrote: [0] https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md This document provide no reasoning about what usecases it supports: Is it possible to create objects that are shared just for short periods during their livetime and guarantee that they can be used threadsave like Manu want it to be? Does it prohibit misuse any better than Manus proposal (that requires the "Expert" to implement all theadsave API)? Is the "normal" User still enforced to do some unsave casts? Has the normal User to have high knownledge of how a threadsave API is to be used or can the compiler provide any guarantees that using them can only fail if the implementation behind the API has bugs (e.g. provide some encapsulation)? Or any other usecases why and how this design is better than what we have now? And also some ideas how to implement some useacases (examples) are completely missing.
Re: shared - i need it to be useful
On 10/17/2018 12:20 AM, Manu wrote: What does it mean 'aliased' precisely? Aliasing means there are two paths to the same piece of data. That could be two pointers pointing to the same data, or one pointer to a variable that is accessible by name. It doesn't really give us anything in practice that we don't have in C++. It provides a standard, enforced way to distinguish shared data from unshared data, and no way to bypass it in @safe code. There's no way to do that in C++.
Re: shared - i need it to be useful
On 19/10/2018 9:02 PM, Walter Bright wrote: On 10/17/2018 4:29 AM, jmh530 wrote: Isn't that also true for isolated data (data that only allows one alias)? That's colloquially called "unique" data. And yes, it is also true for that. That's why casting the return value of malloc() to 'shared' is safe. It's just that the language has no way to semantically identify unique data with its current type system. Actually we kind of have a way to do this (I think). Scope. It can't cross the thread boundary and it has a pretty clear owner as far as the stack is concerned. Given a bit of time and empowering of scope, we could do a @scopeonly type attribute for functions.
Re: shared - i need it to be useful
On 10/17/2018 4:29 AM, jmh530 wrote: Isn't that also true for isolated data (data that only allows one alias)? That's colloquially called "unique" data. And yes, it is also true for that. That's why casting the return value of malloc() to 'shared' is safe. It's just that the language has no way to semantically identify unique data with its current type system.
Re: shared - i need it to be useful
On 19/10/2018 7:09 PM, Norm wrote: There's another way; Stanislav isn't one you need to convince so if that particular discussion is unproductive and disruptive just ignore it. I.e technical discussions should be robust but once they become personal just ignore that input and move on. Isn't always possible I know but in this case I reckon you can. Trust me, I think a few of us at least have already figured that out. Convincing Walter, Andrei and the rest of the core dev team of course will require a DIP. Keep going on this, it is the first hint of movement with shared since like foreva! As long as it doesn't look like my idea[0] (Andrei doesn't like it, I may have asked) it should have some sort of legs. [0] https://github.com/rikkimax/DIPs/blob/shared/DIPs/DIP1xxx-RC2.md
Re: shared - i need it to be useful
On Friday, 19 October 2018 at 02:20:22 UTC, Manu wrote: On Thu., 18 Oct. 2018, 7:10 pm Stanislav Blinov via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: On Friday, 19 October 2018 at 01:53:00 UTC, Manu wrote: > This is a red-herring. > In short, he made up this issue, it doesn't exist. > This is just hot air, and only strengthen my conviction. >> Produce, or drop this presumptious crap. > You are an obscene person. I'm out. Oooh, I'm srry, come baack! Really though, what is it that you wanted to achieve here? You ask for counter-arguments, are given them on *17 pages already*, are asked numerous times to actually demonstrate the value of a small contained portion of your proposal, and all you do is shrug this all off just because you presume to "know better", and on top of that have the audacity to call someone else *obscene*? Wow... just... wow! > You win. I didn't know it was a contest. I've given use cases constantly, about taking object ownership, promotions, and distribution for periods (think parallel for), I can achieve all my goals with full @safety, absolutely no casts in user code, and I have infrastructure in production that applies these patterns successfully. It's worth pursuing. I've spent years thinking on this, I'm trying to move the needle on this issue for the first time in over a decade at least, and you have violently opposed, in principle, from the very first post, and make no effort to actually understand the proposition. It's clearly a contest from your insurance that my proposal in worthless in every single post you've made. You want me to admit defeat and desist. Fuck you. You win. I don't have the time or energy to argue against a wall. You are obscene, you're complete unproductive, and destructive from no apparent reason. I hope you continue to love shared, just the way it is... useless. There's another way; Stanislav isn't one you need to convince so if that particular discussion is unproductive and disruptive just ignore it. I.e technical discussions should be robust but once they become personal just ignore that input and move on. Isn't always possible I know but in this case I reckon you can. Convincing Walter, Andrei and the rest of the core dev team of course will require a DIP. Keep going on this, it is the first hint of movement with shared since like foreva! bye, norm bye, Norm
Re: shared - i need it to be useful
On Friday, 19 October 2018 at 02:20:22 UTC, Manu wrote: I've given use cases constantly, about taking object ownership, promotions, and distribution for periods (think parallel for), Manu, you haven't shown *any* code in which conversion from mutable to shared, an *implicit* one at that, was even present, let alone useful. While at the same time blatantly dismissing *all* examples to the contrary as "bad code" or any other reason, especially the "you just don't understand". How do you expect us to understand if all you do is talk about how it's useful, but don't explain why? You can talk and brag all you want, but until you do show at least one example, I'm sorry but you have no case. And what's this all about? A *small*, localized portion of your proposal, that isn't event the source of the problems you're talking about in the current state of affairs. In all examples flying about in this thread, all of *required* casts were *the opposite*, from shared to mutable, and you yourself acknowledge that those are indeed required on the lowest level. As I've said numerous times, from all that's been said and implied in this thread, the *only* use of implicit casting that comes to mind is avoiding writing some forwarding methods, that's about it. I honestly can't see how this is valuable to warrant such a drastic change in the language. Now maybe it's that I'm dumb, then just say so already and we'll all move on. Just don't act like you're the only one who knows it all and everyone else is an unwashed pleb clinging to worshiping the faces in the trees. I can achieve all my goals with full @safety, absolutely no casts in user code, and I have infrastructure in production that applies these patterns successfully. It's worth pursuing. Okay... Then please do show *one* example of useful implicit conversion from mutable to shared. I've spent years thinking on this, I'm trying to move the needle on this issue for the first time in over a decade at least, And you're behaving in this thread as if everyone else, myself included, were sitting on our thumbs counting flies for this same decade. and you have violently opposed, in principle, from the very first post, and make no effort to actually understand the proposition. I haven't done such a thing. I have asked you, numerous times, one, and only one question, and you never so much as replied to *that question*. What is the big value of this implicit conversion that it would warrant changing current language rules regarding type conversions? It's clearly a contest from your insurance that my proposal in worthless in every single post you've made. You want me to admit defeat and desist. And here you are, continuing to presume. Now you suddenly know what I want. Marvelous. I *don't* want you to admit any defeat, *or* desist. I *want* you to succeed. I *want* to help make `shared` useful so that I and everyone else can actually start writing code with it, not in spite of it. I've agreed with you on pretty much everything in your proposal, *except one thing*. I want you to demonstrate the practical value of *that thing*, and it's benefits over the current state of affairs, and I asked you several times to explain to us mere unwashed mortals exactly how it's useful. What have we been doing for 17 pages? Discussing the benefits of disabling reads/writes on shared? No. Estimating how much existing code could go to trash, what parts of DRuntime/Phobos would need a rewrite? No. We were in constant back-and-forth of "But... nope... but... nope" about this implicit conversion, which you value so much yet for some reason fail to defend. Saying "I had good time with it" is not a very practical defense, not for a language construct anyway. Fuck you. You win. I don't have the time or energy to argue against a wall. If you ask to destroy, be prepared for a fight. Or don't ask. Just stop appealing to your own authority. You are obscene, you're complete unproductive, and destructive from no apparent reason. Give it all you've got, please. Let it all out all at once. I hope you continue to love shared, just the way it is... useless. Yet another presumption. Good on you.
