Re: Shared - Another Thread
On Sat., 20 Oct. 2018, 7:00 am Stanislav Blinov via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: > On Saturday, 20 October 2018 at 02:09:56 UTC, Dominikus Dittes > Scherkl wrote: > > On Saturday, 20 October 2018 at 00:46:36 UTC, Nicholas Wilson > > wrote: > >> Mutable = value may change > >> const = I will not change the value > >> immutable = the value will not change > >> > >> unshared = I (well the current thread) owns the reference > >> shared = reference not owned, no unordered access, no > >> (unordered) writes > >> threadsafe = ??? > > unshared = the current thread owns the reference > > threadsafe = I guarantee no race conditions or deadlocks will > > occur > > shared = every thread may have references > > Exactly, "thredsafe" in nothing more than a contract between > programmers. > When you have "const" data, it is trivial for the compiler to > enforce that: it just doesn't allow you to mutate it. But the > compiler cannot reason about whether your logic is "threadsafe" > or not, there can be no static enforcement of "thread-safety". It > only holds as an implicit contract between programmers: the > authors of data and functions, and the users of that data and > functions, i.e. the API and the callers. > Only at the level of the trusted functions. It is *very* easy to write a correct Atomic implementation. Queues and stuff are well understood and have great reference implementations. If you don't write @trusted functions (most wouldn't!), then you can't mess up. >
Re: Shared - Another Thread
On Saturday, 20 October 2018 at 02:09:56 UTC, Dominikus Dittes Scherkl wrote: On Saturday, 20 October 2018 at 00:46:36 UTC, Nicholas Wilson wrote: Mutable = value may change const = I will not change the value immutable = the value will not change unshared = I (well the current thread) owns the reference shared = reference not owned, no unordered access, no (unordered) writes threadsafe = ??? unshared = the current thread owns the reference threadsafe = I guarantee no race conditions or deadlocks will occur shared = every thread may have references Exactly, "thredsafe" in nothing more than a contract between programmers. When you have "const" data, it is trivial for the compiler to enforce that: it just doesn't allow you to mutate it. But the compiler cannot reason about whether your logic is "threadsafe" or not, there can be no static enforcement of "thread-safety". It only holds as an implicit contract between programmers: the authors of data and functions, and the users of that data and functions, i.e. the API and the callers.
Re: Shared - Another Thread
On Sat., 20 Oct. 2018, 12:10 am Dominikus Dittes Scherkl via Digitalmars-d, wrote: > On Saturday, 20 October 2018 at 06:04:45 UTC, Manu wrote: > > How can you find that such a construct carries its weight with > > respect > > to its rare-ness, when its usefulness is very limited to begin > > with? > > I suggested it only because of the resistance to the proposed > implicit cast to shared. But I agree - a cast from mutable to > immutable could also be implicit, and would rarely cause any > problems. Still, I'm sure you would face equally strong > resistance against that. > If an implicit cast from mutable to immutable existed, then immutable would just be const ;) >
Re: Shared - Another Thread
On Sat., 20 Oct. 2018, 12:10 am Dominikus Dittes Scherkl via Digitalmars-d, wrote: > On Saturday, 20 October 2018 at 06:04:45 UTC, Manu wrote: > > How can you find that such a construct carries its weight with > > respect > > to its rare-ness, when its usefulness is very limited to begin > > with? > > I suggested it only because of the resistance to the proposed > implicit cast to shared. But I agree - a cast from mutable to > immutable could also be implicit, and would rarely cause any > problems. Still, I'm sure you would face equally strong > resistance against that. > It is necessary. My dream of a @safe shared is immaterial without that single important detail. I can't make use of shared without it, at least, not safely, and then we might as well all just pack up and go home. >
Re: Shared - Another Thread
On Saturday, 20 October 2018 at 06:04:45 UTC, Manu wrote: How can you find that such a construct carries its weight with respect to its rare-ness, when its usefulness is very limited to begin with? I suggested it only because of the resistance to the proposed implicit cast to shared. But I agree - a cast from mutable to immutable could also be implicit, and would rarely cause any problems. Still, I'm sure you would face equally strong resistance against that.
Re: Shared - Another Thread
On Fri, Oct 19, 2018 at 5:05 PM Dominikus Dittes Scherkl via Digitalmars-d wrote: > > Therefore it is possible to implicitly cast from mutable or > immutable to const but not in any other direction. > > I think for unshared, shared and threadsave it should be the same: > The second is a declaration attribute, the third a parameter > attribute. I certainly had this thought... if you take a purist point of view, it sounds reasonable. BUT, I think it's a bad choice for practical reasons: 1. Adding a new attribute is heavy handed. 2. It's also unnecessary; the one that would be analogous to immutable is not actually very interesting. At best, it offers the potential for a small number of optimisations, which can be accessed without it. 3. Threadsafety carries a VERY small surface area in the language and in code in general... we don't need to add a pile of machinery for this. Sure, immutable is kinda nice, but in strict terms, it's unnecessary... we could all get on without it. If immutable were removed from the language completely, it would barely affect me. We couldn't get on without const. The same applies here, we must have the const analogous one. Can't live without that. Consider the frequency of immutable to const in any code you've ever seen. This is a very real view of the relative usefulness of the 2 constructs. Now imagine the same comparison to shared... you hardly see shared at all to begin with; and the one analogous to immutable would be relatively so much more rare still. How can you find that such a construct carries its weight with respect to its rare-ness, when its usefulness is very limited to begin with?
Re: Shared - Another Thread
On Fri, Oct 19, 2018 at 4:45 PM Dominikus Dittes Scherkl via Digitalmars-d wrote: > > On Friday, 19 October 2018 at 18:11:50 UTC, Manu wrote: > > On Fri, Oct 19, 2018 at 6:45 AM Dominikus Dittes Scherkl via > > Digitalmars-d wrote: > >> > >> On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: > >> > [...] What issues am I failing to address? > >> [...] Another point is the part of "how can the compiler > >> support the expert in writing threadsave methods" - which you > >> answered with "not a little bit at the moment, but we may > >> improve this in the future" - and that is not at all > >> satisfying. > > > > I think you've misunderstood. > > My proposal is @safe... if you stay @safe, you will receive > > guarantee that your code is threadsafe. > On user side, yes. > > If you want to implement a low-level device, you must implement > > a @trusted function, and I don't know what the compiler can do > > to help you. > Yes, but that's seldom. More often the "expert" will write new > shared types using the low level trusted functions like anybody > else. But that > still requires special care - he has to consider tread-safety in > every > method of a new type, even the non-shared ones. And he has to > fill any > possible gap like construction and assignment so that the > end-user is really sure to not accidentally misusing the type! > And I think a serious proposal need to address this - I think the > compiler could really help here (e.g. prescribe what operators > need to be overloaded and check that all methods use the proper > mechanisms to lock the shared members before operating on them > etc.) > > > So saying that my response that "there is @trusted code at the > > bottom of the stack" is not satisfying is really just a comment > > on your opinion about @trusted code in general. > That just comes on top of it. > > > My proposal is designed to be useful and @safe for *users* as > > primary goal. > I agree with you, but others seem not so convinced (yet?). > [...] > > The user has manually cast to unshared inside their > > unsafe/(@trusted?) function, what more signal do they need that > > they've engaged in an unsafe operation? > Some hints what to do to be able to trust them? > You asked what issues you were failing to address. That was just > some ideas of mine what you may address in addition. I understand. I don't have good ideas to add mechanical guarantees, other than something extremely likely to flag false-positives like "any `shared` piece of data involved in an unsafe cast in any function should look suspicious when accessed elsewhere within this module"...? I think this is an area for further development, but I don't think it's a barrier to making shared a useful @safe language construct with respect to the type safety. It is possible to define the typesafety rules correctly and then start writing experimental code. > More often the "expert" will write new > shared types using the low level trusted functions like anybody > else. But that > still requires special care - he has to consider tread-safety in > every > method of a new type, even the non-shared ones. This shouldn't be true unless they're writing new @trusted functions. If they're staying @safe, then there is nothing you can do to the lower-level utility that's not threadsafe, so you can freely use it throughout your new type. If this is true, then the @trusted function is not threadsafe as it promises. One thing I know is, your proposition above is unusual. In my experience among hundreds, perhaps thousands of colleagues, people don't just start presuming to write threadsafe tooling. We have 4-5 such threadsafe tools in our library, they are at the bottom of the stack, and would contain @trusted functions under my proposal... all other code just makes use of them (and quite extensive use of them!). People don't write new thredsafe machinery for fun, it's a very serious endeavour. The countless uses of those tools and the structure built on top should be @safely interactible, which is possible under my proposal.
Re: Shared - Another Thread
On Saturday, 20 October 2018 at 00:46:36 UTC, Nicholas Wilson wrote: Mutable = value may change const = I will not change the value immutable = the value will not change unshared = I (well the current thread) owns the reference shared = reference not owned, no unordered access, no (unordered) writes threadsafe = ??? unshared = the current thread owns the reference threadsafe = I guarantee no race conditions or deadlocks will occur shared = every thread may have references
Re: Shared - Another Thread
On Saturday, 20 October 2018 at 00:00:49 UTC, Dominikus Dittes Scherkl wrote: Hmm. mutable, immutable and const form a triple, the second is a declaration attribute, the last an parameter attribute, indicating that you don't want to modify the parameter, may it be because you can't (as it is immutable) or you only don't need to despite it would be possible (if it was mutable). The later is your responsibility to guarantee (with the help from the compiler). Therefore it is possible to implicitly cast from mutable or immutable to const but not in any other direction. I think for unshared, shared and threadsave it should be the same: The second is a declaration attribute, the third a parameter attribute. The first two can implicitly be cast to threadsave, may be because it is thread-local and therefore no race condition is possible, or may be because you take special care in your type to guarantee the thread safety by using atomic operations or locking or whatever. That make it possible, that the implicit cast from shared to unshared can be avoided while still providing functions that can take both kinds of arguments. Yes, that would add a little to the attribute bloat (new keyword) but not to the number of attributes per type or parameter. Mutable = value may change const = I will not change the value immutable = the value will not change unshared = I (well the current thread) owns the reference shared = reference not owned, no unordered access, no (unordered) writes threadsafe = ???
Re: Shared - Another Thread
On Friday, 19 October 2018 at 15:46:20 UTC, Stanislav Blinov wrote: On Friday, 19 October 2018 at 13:40:54 UTC, Dominikus Dittes Scherkl wrote: Conflating "shared" and "threadsave" in that manner was, I think, the biggest mistake of your proposal. He talked about it in a previous thread, and generally I would agree with him that such conflation is indeed beneficial provided that some concessions are made for `shared`. Moreover, yet another attribute? Please no... Hmm. mutable, immutable and const form a triple, the second is a declaration attribute, the last an parameter attribute, indicating that you don't want to modify the parameter, may it be because you can't (as it is immutable) or you only don't need to despite it would be possible (if it was mutable). The later is your responsibility to guarantee (with the help from the compiler). Therefore it is possible to implicitly cast from mutable or immutable to const but not in any other direction. I think for unshared, shared and threadsave it should be the same: The second is a declaration attribute, the third a parameter attribute. The first two can implicitly be cast to threadsave, may be because it is thread-local and therefore no race condition is possible, or may be because you take special care in your type to guarantee the thread safety by using atomic operations or locking or whatever. That make it possible, that the implicit cast from shared to unshared can be avoided while still providing functions that can take both kinds of arguments. Yes, that would add a little to the attribute bloat (new keyword) but not to the number of attributes per type or parameter.
Re: Shared - Another Thread
On Friday, 19 October 2018 at 18:11:50 UTC, Manu wrote: On Fri, Oct 19, 2018 at 6:45 AM Dominikus Dittes Scherkl via Digitalmars-d wrote: On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: > [...] What issues am I failing to address? [...] Another point is the part of "how can the compiler support the expert in writing threadsave methods" - which you answered with "not a little bit at the moment, but we may improve this in the future" - and that is not at all satisfying. I think you've misunderstood. My proposal is @safe... if you stay @safe, you will receive guarantee that your code is threadsafe. On user side, yes. If you want to implement a low-level device, you must implement a @trusted function, and I don't know what the compiler can do to help you. Yes, but that's seldom. More often the "expert" will write new shared types using the low level trusted functions like anybody else. But that still requires special care - he has to consider tread-safety in every method of a new type, even the non-shared ones. And he has to fill any possible gap like construction and assignment so that the end-user is really sure to not accidentally misusing the type! And I think a serious proposal need to address this - I think the compiler could really help here (e.g. prescribe what operators need to be overloaded and check that all methods use the proper mechanisms to lock the shared members before operating on them etc.) So saying that my response that "there is @trusted code at the bottom of the stack" is not satisfying is really just a comment on your opinion about @trusted code in general. That just comes on top of it. My proposal is designed to be useful and @safe for *users* as primary goal. I agree with you, but others seem not so convinced (yet?). [...] The user has manually cast to unshared inside their unsafe/(@trusted?) function, what more signal do they need that they've engaged in an unsafe operation? Some hints what to do to be able to trust them? You asked what issues you were failing to address. That was just some ideas of mine what you may address in addition.
Re: Shared - Another Thread
On Fri, Oct 19, 2018 at 6:45 AM Dominikus Dittes Scherkl via Digitalmars-d wrote: > > On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: > > On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: > >> What cracks me up with Manu's proposal is that it is its > >> simplicity and lack of ambition that is criticized the most. > >> shared is a clusterfuck, according to what I gathered from the > >> forum, I never had yet to use it in my code. Manu's idea makes > >> it a little less of a clusterfuck, and people attack the idea > >> because it doesn't solve all and everything that's wrong with > >> shared. Funny. > >> > > > > Elaborate on this... It's clearly over-ambitious if anything. > > What issues am I failing to address? > > First of all, you called it "shared", but what your concept > describes is "theadsave". > If you had called it the later, it would have been clear to > everybody that thread local data is indeed automatically > threadsave, because only one thread has access to it (that > "implicit conversion"). But if something is "shared" (in the > common-world sense), it is of course no more "threadsave" - you > have to implement special methods to treat it. > > Conflating "shared" and "threadsave" in that manner was, I think, > the biggest mistake of your proposal. > > Another point is the part of "how can the compiler support the > expert in writing threadsave methods" - which you answered with > "not a little bit at the moment, but we may improve this in the > future" - and that is not at all satisfying. I think you've misunderstood. My proposal is @safe... if you stay @safe, you will receive guarantee that your code is threadsafe. If you want to implement a low-level device, you must implement a @trusted function, and I don't know what the compiler can do to help you. You will have to produce unsafe casts, so at least it will be completely clear every operation you perform that is unsafe. Users of your @trusted library though will experience @safe code upward through the stack. So saying that my response that "there is @trusted code at the bottom of the stack" is not satisfying is really just a comment on your opinion about @trusted code in general. My proposal is designed to be useful and @safe for *users* as primary goal. Authors that are composing low-level tools/libs will enjoy @safe guarantees. Only authors implementing ground-level machinery would need to write @trusted code... and that's the nature of the whole @safe stack. @safe can't practically exist without @trusted at the bottom of the stack. > Are there really no > ideas? I have some ideas, but I don't think they're practical to implement. Some cross-referencing of access would be required for the compiler to suspect foul-play. Functions would need to be analysed in conjunction; that sounds hard to implement. > No check that the proper atomic funtions are used or the > cast to "unshared" is ok at where it is used? The user has manually cast to unshared inside their unsafe/(@trusted?) function, what more signal do they need that they've engaged in an unsafe operation? > Even the expert > needs a little help to find the upcomming and well hidden bugs in > their oh so threadsave API... I think a lot of people probably over-estimate the number of tooling libs that live at the bottom of the stack. The list is fairly short: Atomic, Mutex/Semaphore, and a couple of lock-free-queue/list type structures. I don't know what other useful constructs exist... they will all be in libraries. Users would almost never come in contact with unsafe code, and again, that's the whole point of my design!
Re: Shared - Another Thread
On Friday, 19 October 2018 at 13:40:54 UTC, Dominikus Dittes
Scherkl wrote:
On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote:
On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh
wrote:
What cracks me up with Manu's proposal is that it is its
simplicity and lack of ambition that is criticized the most.
shared is a clusterfuck, according to what I gathered from
the forum, I never had yet to use it in my code. Manu's idea
makes it a little less of a clusterfuck, and people attack
the idea because it doesn't solve all and everything that's
wrong with shared. Funny.
Elaborate on this... It's clearly over-ambitious if anything.
What issues am I failing to address?
First of all, you called it "shared", but what your concept
describes is "theadsave".
If you had called it the later, it would have been clear to
everybody that thread local data is indeed automatically
threadsave, because only one thread has access to it (that
"implicit conversion"). But if something is "shared" (in the
common-world sense), it is of course no more "threadsave" - you
have to implement special methods to treat it.
Conflating "shared" and "threadsave" in that manner was, I
think, the biggest mistake of your proposal.
He talked about it in a previous thread, and generally I would
agree with him that such conflation is indeed beneficial provided
that some concessions are made for `shared`. Moreover, yet
another attribute? Please no...
struct X {
void foo(threadsafe const shared Bar* bar) @nogc @trusted
notrhow pure const shared threadsafe;
}
Attribute explosion is bad enough already.
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: What cracks me up with Manu's proposal is that it is its simplicity and lack of ambition that is criticized the most. shared is a clusterfuck, according to what I gathered from the forum, I never had yet to use it in my code. Manu's idea makes it a little less of a clusterfuck, and people attack the idea because it doesn't solve all and everything that's wrong with shared. Funny. Elaborate on this... It's clearly over-ambitious if anything. What issues am I failing to address? First of all, you called it "shared", but what your concept describes is "theadsave". If you had called it the later, it would have been clear to everybody that thread local data is indeed automatically threadsave, because only one thread has access to it (that "implicit conversion"). But if something is "shared" (in the common-world sense), it is of course no more "threadsave" - you have to implement special methods to treat it. Conflating "shared" and "threadsave" in that manner was, I think, the biggest mistake of your proposal. Another point is the part of "how can the compiler support the expert in writing threadsave methods" - which you answered with "not a little bit at the moment, but we may improve this in the future" - and that is not at all satisfying. Are there really no ideas? No check that the proper atomic funtions are used or the cast to "unshared" is ok at where it is used? Even the expert needs a little help to find the upcomming and well hidden bugs in their oh so threadsave API...
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 21:14:54 UTC, Stanislav Blinov wrote: On Thursday, 18 October 2018 at 20:59:59 UTC, Erik van Velzen wrote: [...] Quite a simple reason: it was years ago, however old you are now you were younger and less experienced, and probably didn't understand something back then. [...] Then I don't know what to tell you. It literally talks about compiler forbidding unsafe operations and *requiring* you to go the extra mile, by just rejecting invalid code (something that Manu is proposing to forego!). But that's *code*, not logic. [...] Tangetially?! There's a whole section on writing `shared`-aware code (none of which would even compile today, I don't know if it's addressed in his errata). [...] Yeah, some of that never happened and never will. But that aside, none of it says "threading will be safe by default". It says "threading will be a lot less unsafe by default". And *that* is what we must achieve. The "threading will be a lot less unsafe by default" is related to the default TLS usage. I remember like Erik, maybe wrongly, that the ambitions on shared were more directed towards the "threading will be safe by default" goal. I've to read again some post from Bartosz Milewski... /Paolo
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 20:59:59 UTC, Erik van Velzen wrote: Let me start by saying I'm willing to admit that I was factually wrong. Also keep in mind that "me having an impression" is something that is can't be independently verified and you'll have to take my at my word. Just that the exact reason for that impression was lost to the sands of time. Quite a simple reason: it was years ago, however old you are now you were younger and less experienced, and probably didn't understand something back then. Your impression was wrong. Open e.g. TDPL and read up on `shared` how it was envisioned back then. I don't think the book really supports your argument. The first paragraph about shared sound to me like "the compiler will automagically fix it". Then I don't know what to tell you. It literally talks about compiler forbidding unsafe operations and *requiring* you to go the extra mile, by just rejecting invalid code (something that Manu is proposing to forego!). But that's *code*, not logic. Only tangentially it is mentioned that you're actually supposed to write special code yourself. You would have to be a compiler expert to draw the correct conclusion. Tangetially?! There's a whole section on writing `shared`-aware code (none of which would even compile today, I don't know if it's addressed in his errata). Also the last paragraph the quote below is interesting in light of our other discussion about casting to shared. From the book: [snip] Yeah, some of that never happened and never will. But that aside, none of it says "threading will be safe by default". It says "threading will be a lot less unsafe by default". And *that* is what we must achieve.
Re: Shared - Another Thread
Let me start by saying I'm willing to admit that I was factually wrong. Also keep in mind that "me having an impression" is something that is can't be independently verified and you'll have to take my at my word. Just that the exact reason for that impression was lost to the sands of time. On Thursday, 18 October 2018 at 20:13:49 UTC, Stanislav Blinov wrote: On Thursday, 18 October 2018 at 20:10:18 UTC, Erik van Velzen wrote: When shared stood up in its current form, expectation was made "this will be threadsafe automatically - we'll figure out how in the future". It never was like that. At all. I don't think either Walter or Andrei are idiots, do you? Obviously not. But they are not infallible and can also change their mind. And features can be used beyond their initially envisioned purpose. Because it works for global variables. But it doesn't seem like an expectation we can deliver on. (I have no direct reference to this but that was certainly my impression) Your impression was wrong. Open e.g. TDPL and read up on `shared` how it was envisioned back then. I don't think the book really supports your argument. The first paragraph about shared sound to me like "the compiler will automagically fix it". Only tangentially it is mentioned that you're actually supposed to write special code yourself. You would have to be a compiler expert to draw the correct conclusion. Also the last paragraph the quote below is interesting in light of our other discussion about casting to shared. From the book: 13.11 The shared Type Qualifier (...) To the type system, shared indicates that several threads have access to a piece of data. The compiler acknowledges that reality by restricting operations on shared data and by generating special code for the accepted operations. (...) Because all shared data is accounted for and protected under the aegis of the language, passing shared data via send and receive is allowed. (...) 13.12 Operations with shared Data and Their Effects Working with shared data is peculiar because multiple threads may read and write it at any moment. Therefore, the compiler makes sure that all operations preserve integrity of data and also causality of operations. Reads and writes of shared values are allowed and guaranteed to be atomic: nu- meric types (save for real), pointers, arrays, function pointers, delegates, and class ref- erences. struct types containing exactly one of the mentioned types are also readable and writable atomically. (...) For all numeric types and function pointers, shared-qualified values are convertible implicitly to and from unqualified values. Pointer conversions between shared(T*) and shared(T)* are allowed in both directions. Primitives in std.concurrency allow you to do arithmetic on shared numeric types.
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 19:09:42 UTC, Patrick Schluter wrote: On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: Elaborate on this... It's clearly over-ambitious if anything. What issues am I failing to address? I'm creating a situation where using shared has a meaning, is safe, and doesn't require any unsafe interactions, no casts, etc, for users at any level above the bare metal tooling... How would you improve on that proposition? No, your proposition is not the issue here. The problem I see is the expectation people have with what shared is supposed to do. I have the impression from reading in this forum about shared that people expect that just putting a shared in front of a variable will solve all the concurrency problems in existance. I think you hit the nail on the head here. When shared stood up in its current form, expectation was made "this will be threadsafe automatically - we'll figure out how in the future". Because it works for global variables. But it doesn't seem like an expectation we can deliver on. (I have no direct reference to this but that was certainly my impression) Your proposition doesn't want to address this utopic goal and that is a good thing imo. Adding that restriction that you propose makes explicit what was implied but not clearly stated until now. I'm not good enough in D to add more than a meta reflexion on the subject so I will not follow up on that. I often have the impression that a lot of things are going slower than necessary because a mentality where the perfect is in the way of good.
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 20:10:18 UTC, Erik van Velzen wrote: When shared stood up in its current form, expectation was made "this will be threadsafe automatically - we'll figure out how in the future". It never was like that. At all. I don't think either Walter or Andrei are idiots, do you? Because it works for global variables. But it doesn't seem like an expectation we can deliver on. (I have no direct reference to this but that was certainly my impression) Your impression was wrong. Open e.g. TDPL and read up on `shared` how it was envisioned back then.
Re: Shared - Another Thread
On Thu, Oct 18, 2018 at 07:09:42PM +, Patrick Schluter via Digitalmars-d wrote: [...] > I often have the impression that a lot of things are going slower than > necessary because a mentality where the perfect is in the way of good. That is indeed an all-too-frequent malady around these parts, sad to say. Which has the sad consequence that despite all efforts, there are still unfinished areas in D, and promises that haven't materialized in years (like multiple alias this). Still, the parts of D that are working well form a very powerful and comfortable-to-use language. Not quite the ideal we wish it to be, but IMO much closer than any other language I've seen yet. Recently I began dabbling in Android programming, and the one thing that keeps sticking out to me is how painful writing Java is. Almost every day of writing Java code has me wishing for this or that feature in D. Slices. Closures. Meta-programming. I found most of my time spent fighting with language limitations rather than make progress with the problem domain. Eventually I resorted to generating Java code from D for some fo the most painful repetitive parts, and the way things are looking, I'm likely to be doing a lot more of that. I fear the way things are going will have be essentially writing a D to Java compiler at some point! D may not be perfect in many ways, but it's still one of the best languages out there right now, IMO. T -- Give me some fresh salted fish, please.
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 17:01:46 UTC, Stanislav Blinov wrote: On Thursday, 18 October 2018 at 16:31:33 UTC, Vijay Nayar wrote: Imagine a simple algorithm that does logic on very long numbers, split into bytes. One multi-threaded implementation may use 4 threads. The first operating on bytes 0, 4, 8, etc. The second operating on bytes 1, 5, 9, etc. In this case, a mutex or lock isn't actually needed, because the algorithm itself assures that threads don't collide. Yes, they do collide. You just turned your cache into a giant clusterf**k. Keyword: MESIF. In that case partitioning in cache line sizes is the least that has to be done.
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 16:24:39 UTC, Manu wrote: On Thu., 18 Oct. 2018, 5:05 am Patrick Schluter via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: >> If something might be used by someone else it's better not >> to touch it, unless one has confirmation it is not used by >> someone else. >> >> This is what shared has to enforce. > > Yes. But how can the compiler statically verify this? > Because if it cannot be statically verified, then somewhere > along the line we have to trust the programmer. Ergo, it's > programming by convention, and we all know how effective > that is. > and that is exactly what shared is currently doing. Adding the rw restriction at least adds a protection for inadvertantly changing a shared object, a thing that doesn't exist now. What cracks me up with Manu's proposal is that it is its simplicity and lack of ambition that is criticized the most. shared is a clusterfuck, according to what I gathered from the forum, I never had yet to use it in my code. Manu's idea makes it a little less of a clusterfuck, and people attack the idea because it doesn't solve all and everything that's wrong with shared. Funny. Elaborate on this... It's clearly over-ambitious if anything. What issues am I failing to address? I'm creating a situation where using shared has a meaning, is safe, and doesn't require any unsafe interactions, no casts, etc, for users at any level above the bare metal tooling... How would you improve on that proposition? No, your proposition is not the issue here. The problem I see is the expectation people have with what shared is supposed to do. I have the impression from reading in this forum about shared that people expect that just putting a shared in front of a variable will solve all the concurrency problems in existance. Your proposition doesn't want to address this utopic goal and that is a good thing imo. Adding that restriction that you propose makes explicit what was implied but not clearly stated until now. I'm not good enough in D to add more than a meta reflexion on the subject so I will not follow up on that. I often have the impression that a lot of things are going slower than necessary because a mentality where the perfect is in the way of good.
Re: Shared - Another Thread
Pardon the snarkiness, I probably need to get some air from that other shared thread.
Re: Shared - Another Thread
On Thursday, 18 October 2018 at 16:31:33 UTC, Vijay Nayar wrote: Imagine a simple algorithm that does logic on very long numbers, split into bytes. One multi-threaded implementation may use 4 threads. The first operating on bytes 0, 4, 8, etc. The second operating on bytes 1, 5, 9, etc. In this case, a mutex or lock isn't actually needed, because the algorithm itself assures that threads don't collide. Yes, they do collide. You just turned your cache into a giant clusterf**k. Keyword: MESIF.
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 21:12:49 UTC, Stefan Koch wrote: Hi, reading the other shared thread "shared - i need to be useful"(https://forum.dlang.org/thread/mailman.4299.1539629222.29801.digitalmar...@puremagic.com) let me to an important realisation concerning the reason shareding data across threads is so unintuitve and hard to get right. The reason is that sharing in the real world has nothing to do with using something and the same time. For example: If I share my flat with another person, that person, while occupying the same flat as me, cannot actually occupy the same space. It is physically impossible. In other words sharing does not mean for multiple entities to own something it's rather about diving and managing the (temporary) ownership of fragments. Therefore if ownership is unclear sharing is impossible. The safest default for something shared with unclear ownership is to view it as untouchable/unreadble/unwritable until ownership is established. My understanding is that the "shared" keyword can be useful especially with array types that are operated on by multiple threads. Some algorithms put together data following specific rules on how that data can be fragmented. Imagine a simple algorithm that does logic on very long numbers, split into bytes. One multi-threaded implementation may use 4 threads. The first operating on bytes 0, 4, 8, etc. The second operating on bytes 1, 5, 9, etc. In this case, a mutex or lock isn't actually needed, because the algorithm itself assures that threads don't collide. It's an over-simplification, but I think this is basically what the prime-number finding algorithm by Jabari Zakiya is doing.
Re: Shared - Another Thread
On Thu., 18 Oct. 2018, 5:05 am Patrick Schluter via Digitalmars-d, < digitalmars-d@puremagic.com> wrote: > On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: > >> If something might be used by someone else it's better not to > >> touch it, unless one has confirmation it is not used by > >> someone else. > >> > >> This is what shared has to enforce. > > > > Yes. But how can the compiler statically verify this? Because > > if it cannot be statically verified, then somewhere along the > > line we have to trust the programmer. Ergo, it's programming by > > convention, and we all know how effective that is. > > > and that is exactly what shared is currently doing. Adding the rw > restriction at least adds a protection for inadvertantly changing > a shared object, a thing that doesn't exist now. > > What cracks me up with Manu's proposal is that it is its > simplicity and lack of ambition that is criticized the most. > shared is a clusterfuck, according to what I gathered from the > forum, I never had yet to use it in my code. Manu's idea makes it > a little less of a clusterfuck, and people attack the idea > because it doesn't solve all and everything that's wrong with > shared. Funny. > Elaborate on this... It's clearly over-ambitious if anything. What issues am I failing to address? I'm creating a situation where using shared has a meaning, is safe, and doesn't require any unsafe interactions, no casts, etc, for users at any level above the bare metal tooling... How would you improve on that proposition? >
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: If something might be used by someone else it's better not to touch it, unless one has confirmation it is not used by someone else. This is what shared has to enforce. Yes. But how can the compiler statically verify this? Because if it cannot be statically verified, then somewhere along the line we have to trust the programmer. Ergo, it's programming by convention, and we all know how effective that is. and that is exactly what shared is currently doing. Adding the rw restriction at least adds a protection for inadvertantly changing a shared object, a thing that doesn't exist now. What cracks me up with Manu's proposal is that it is its simplicity and lack of ambition that is criticized the most. shared is a clusterfuck, according to what I gathered from the forum, I never had yet to use it in my code. Manu's idea makes it a little less of a clusterfuck, and people attack the idea because it doesn't solve all and everything that's wrong with shared. Funny.
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 21:55:48 UTC, H. S. Teoh wrote: The problem, of course, is that they are also charged particles, and the electromagnetic forces that hold the atom in place would be greatly disturbed if two atoms were to occupy the same space simultaneously, leading to a (very fast and very violent) reorganization of nucleii and electrons. What that looks like macroscopically, I can't say exactly, but certainly delicate structures like proteins, DNA, lipid layers, and such would cease to exist, their constituent particles being violently scattered every which way in the course of reorganizing themselves into new structures that would bring the electromagnetic forces back into balance (and that, in all likelihood, won't resemble anything close to their starting molecular structures). Whatever the result may be, I'm pretty certain it would not have good consequences for the biological processes built upon said delicate structures. To say the least. :-D Even worst than that: conversion to/from E is involved in the process! :-P
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 22:56:26 UTC, H. S. Teoh wrote: On Wed, Oct 17, 2018 at 10:13:37PM +, Stefan Koch via Digitalmars-d wrote: On Wednesday, 17 October 2018 at 21:55:48 UTC, H. S. Teoh wrote: [...] > But nobody will be building a fusion engine out of race > conditions anytime in the foreseeable future. :-D [...] It's still valid though. If something might be used by someone else it's better not to touch it, unless one has confirmation it is not used by someone else. This is what shared has to enforce. Yes. But how can the compiler statically verify this? Because if it cannot be statically verified, then somewhere along the line we have to trust the programmer. Ergo, it's programming by convention, and we all know how effective that is. [ or it will lead] to an overly-restrictive conservative type system that nobody will use. I'd rather wear 5 virtual straight-jackets than spending the rest of my life trying to observe invisble races, and wearing a real one. Stefan
Re: Shared - Another Thread
On Wed, Oct 17, 2018 at 10:13:37PM +, Stefan Koch via Digitalmars-d wrote: > On Wednesday, 17 October 2018 at 21:55:48 UTC, H. S. Teoh wrote: [...] > > But nobody will be building a fusion engine out of race conditions > > anytime in the foreseeable future. :-D [...] > Now my analogy sounds silly. Silly or not, think of the (fictional) possibilities if we *could* build fusion engines out of race conditions! We could power starships with a team of incompetent programmers. It would revolutionize everything! > It's still valid though. > > If something might be used by someone else it's better not to touch > it, unless one has confirmation it is not used by someone else. > > This is what shared has to enforce. Yes. But how can the compiler statically verify this? Because if it cannot be statically verified, then somewhere along the line we have to trust the programmer. Ergo, it's programming by convention, and we all know how effective that is. But I'm not sure if it's possible for the compiler to verify it, unless the language itself has a well-defined concurrency model with well-defined semantics. Otherwise the best you could do is to implement heuristics, which will either lead to holes in the type system, or to an overly-restrictive conservative type system that nobody will use. T -- An elephant: A mouse built to government specifications. -- Robert Heinlein
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 21:55:48 UTC, H. S. Teoh wrote: Nah, that's not even anywhere close to nuclear fusion. The atoms which make up your body (and basically everything else) are mostly empty, with just a tiny speck of a nucleus, and a bunch of extremely tiny electrons zipping about. There's plenty of room for hundreds, if not thousands or millions, of persons to occupy the same space without any of the particles running into each other. But nobody will be building a fusion engine out of race conditions anytime in the foreseeable future. :-D T Now my analogy sounds silly. It's still valid though. If something might be used by someone else it's better not to touch it, unless one has confirmation it is not used by someone else. This is what shared has to enforce.
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 21:55:48 UTC, H. S. Teoh wrote: But nobody will be building a fusion engine out of race conditions anytime in the foreseeable future. :-D We should be so blessed...
Re: Shared - Another Thread
On Wed, Oct 17, 2018 at 09:29:07PM +, Stefan Koch via Digitalmars-d wrote: > On Wednesday, 17 October 2018 at 21:12:49 UTC, Stefan Koch wrote: > > [another person] cannot actually occupy the same space. It is > > physically impossible. > > Actually, that's not quite true, If they were to try hard enough the > result would be nuclear fusion, (I am guessing (I am not a phsysist)), > in any case it would certainly mess up the state of everyone involved; > which is exactly what happens win multi-threaded situations. Nah, that's not even anywhere close to nuclear fusion. The atoms which make up your body (and basically everything else) are mostly empty, with just a tiny speck of a nucleus, and a bunch of extremely tiny electrons zipping about. There's plenty of room for hundreds, if not thousands or millions, of persons to occupy the same space without any of the particles running into each other. The problem, of course, is that they are also charged particles, and the electromagnetic forces that hold the atom in place would be greatly disturbed if two atoms were to occupy the same space simultaneously, leading to a (very fast and very violent) reorganization of nucleii and electrons. What that looks like macroscopically, I can't say exactly, but certainly delicate structures like proteins, DNA, lipid layers, and such would cease to exist, their constituent particles being violently scattered every which way in the course of reorganizing themselves into new structures that would bring the electromagnetic forces back into balance (and that, in all likelihood, won't resemble anything close to their starting molecular structures). Whatever the result may be, I'm pretty certain it would not have good consequences for the biological processes built upon said delicate structures. To say the least. :-D Two threads trying to access the same memory location at the same time, by comparison, would tend to have much less drastic, though no less dire, consequences. :-D Program crashes and security holes aside, at least we aren't talking about the transistors in the CPU suddenly (and violently) rearranging themselves into an unrecognizable slag (possibly with accompanying loud noises and flying shrapnel). Even the program structure would mostly remain intact, though of course, logical consistency would be compromised, and you know what Walter says about programs that continue bungling forwards after a false assertion -- the code might wind up doing arbitrary things it was never intended to do, while being perfectly convinced that it is doing exactly what it was told. But nobody will be building a fusion engine out of race conditions anytime in the foreseeable future. :-D T -- All men are mortal. Socrates is mortal. Therefore all men are Socrates.
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 21:40:35 UTC, Stanislav Blinov wrote: Now, I perfectly understand what Manu wants: for `shared` to stop being a stupid keyword that nobody uses, and start bringing in value to the language. At the moment, the compiler happily allows you to write and read `shared` unhindered, which isn't useful at all. Very Good, that's what I wanted to happen :)
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 21:29:07 UTC, Stefan Koch wrote: in any case it would certainly mess up the state of everyone involved; which is exactly what happens win multi-threaded situations. ^ that is very true. And that is why: - one must not keep shared and local data close together (e.g. within same cache line) - one must not implicitly convert local data to shared data Now, I perfectly understand what Manu wants: for `shared` to stop being a stupid keyword that nobody uses, and start bringing in value to the language. At the moment, the compiler happily allows you to write and read `shared` unhindered, which isn't useful at all. It also allows you to have weird things like shared destructors and postblits (which got extended to whole shared copy ctors in a DIP!). Latter is especially painful when attempting to define the whole type `shared`.
Re: Shared - Another Thread
On Wednesday, 17 October 2018 at 21:12:49 UTC, Stefan Koch wrote: [another person] cannot actually occupy the same space. It is physically impossible. Actually, that's not quite true, If they were to try hard enough the result would be nuclear fusion, (I am guessing (I am not a phsysist)), in any case it would certainly mess up the state of everyone involved; which is exactly what happens win multi-threaded situations.
