Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/22/17 12:15 PM, Kagamin wrote: https://issues.dlang.org/show_bug.cgi?id=6585 is this fixed too? How various opIndex will behave now? Seems to work. I closed as a duplicate. -Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
https://issues.dlang.org/show_bug.cgi?id=6585 is this fixed too? How various opIndex will behave now?
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/16/17 11:23 AM, Steven Schveighoffer wrote: On 8/16/17 8:58 AM, Steven Schveighoffer wrote: However, I have found a better way to call postblit that involves the qualifiers than the way Variant currently does it. I'm going to submit a PR to fix these issues. https://github.com/dlang/phobos/pull/5694 This has been merged, so you should now be able to send shared types properly through send/receive on master dmd. Don't think it made it into 2.076 beta though. -Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 21 August 2017 at 02:17:57 UTC, crimaniak wrote: ... shared A a; ... Sorry, accidental delete, read this as shared A a = new shared(A);
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:
And no, this isn't ideal, but the only semi-decent solution
that's been proposed that safely casts away shared for you is
synchronized classes, which Andrei describes in TDPL but have
never been implemented.
After reading this I did some experiment to understand the
situation better. I make a simple class and unittest:
// dmd sync1.d -unittest -main
unittest
{
import std.stdio;
synchronized
class A
{
private int a;
void inc()
{
++a;
}
int get(){ return a;}
}
shared A a;
for(int i=0; i<100; ++i)
a.inc();
writeln(a.get);
}
Oops! Deprecation: read-modify-write operations are not allowed
for shared variables. Use core.atomic.atomicOp!"+="(this.a, 1)
instead.
Why use atomic operations if the class already synchronized?
Well..
...
import core.atomic: atomicOp;
...
// ++a; // Deprecation: read-modify-write operations are not
allowed for shared variables. Use
core.atomic.atomicOp!"+="(this.a, 1) instead.
atomicOp!"+="(this.a, 1);
...
ok, works. But it works by the way as if synchronized just makes
all methods shared, but does not provide the object methods with
a mutex lock, as Java does. Am I right here? And what preventing
to implement it right, lack of manpower or some ideologic
problems?
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Thursday, 17 August 2017 at 13:09:29 UTC, Kagamin wrote: On Wednesday, 16 August 2017 at 23:15:10 UTC, crimaniak wrote: I wonder if it possible and usable to make some template to support this pattern, where we give mutex(es), shared object(s) and delegate to operate with objects as non-shared. https://dpaste.dzfl.pl/8b3b05c8ec0a like this? Not sure if it helps, don't forget that it's a casted shared object. Yes, something like this. In general, I thought about the possibility of using several shared objects in this block but then realized that everything can be reduced to the case of one object.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/17/17 8:41 AM, Kagamin wrote: On Wednesday, 16 August 2017 at 13:14:55 UTC, Steven Schveighoffer wrote: But that isn't a concern for Variant. It is only calling the postblit, which does work. Shouldn't it call destructor when it goes out of scope? You're right, and it does. It uses the typeid to destroy it, which I think ignores any attributes. I've updated my PR to switch to the __xdtor method, which takes into account attributes of the method. This doesn't suffer from the same incorrect assumption the compiler makes when destroying something in a scope, so ironically, using a Variant to wrap a shared type with a destructor is going to work better than using the stack :) -Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Wednesday, 16 August 2017 at 23:15:10 UTC, crimaniak wrote: I wonder if it possible and usable to make some template to support this pattern, where we give mutex(es), shared object(s) and delegate to operate with objects as non-shared. https://dpaste.dzfl.pl/8b3b05c8ec0a like this? Not sure if it helps, don't forget that it's a casted shared object.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Wednesday, 16 August 2017 at 13:14:55 UTC, Steven Schveighoffer wrote: But that isn't a concern for Variant. It is only calling the postblit, which does work. Shouldn't it call destructor when it goes out of scope?
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:
The way to handle shared is to protect the section of code
that's using the shared object with either a mutex or
synchronized block, and then you cast away shared from the
object within that section and operate on it as thread-local.
When you're done, you make sure that you don't have any
thread-local references to the data, and you release the mutex
or exit the synchronized block. e.g. something like
shared T sharedObj = getSharedObj();
synchronized(mutex)
{
T nonSharedObj = cast(T)sharedObject
// do stuff...
// make sure that no references to nonSharedObj have escaped
}
Casting objects just to work with it every time is so bad style
for me that I even didn't consider such possibility. In fact, I
did make something like this but with __gshared both object and
mutex. I think I need to review this part of site engine and my
programming habits too.
I wonder if it possible and usable to make some template to
support this pattern, where we give mutex(es), shared object(s)
and delegate to operate with objects as non-shared.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Tuesday, 15 August 2017 at 21:54:23 UTC, Steven Schveighoffer
wrote:
On 8/15/17 5:27 PM, Arek wrote:
On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:
Well, no wrapper is actually needed here:
[...]
The issue is that send cannot handle shared value types due to
a bug in the implementation. In essence, there is no reason to
send a shared A -- it's an unrelated copy and not actually
shared.
You can send a shared reference to an A just fine:
static shared A a;
send(tid, ); // works
You *should* be able to send a shared(A). There is no reason to
disallow it. But again, it's not super useful.
Yes, I absolutely agree. The object may be shared due to other
requirements and there is no reason to reject 'share' qualifier
what I want to send/receive it.
This very simple code also doesn't compile:
shared struct S
{
int i;
~this()
{
}
}
void main()
{
shared s = shared S();
}
In general, shared structs with postblit and destructor make
problems.
postblit should work.
It does look like destructors are a problem. It appears the
compiler attempts to call the destructor while unshared.
-Steve
But I have no idea how and when the object loses its shared
nature. Yes, it looks like a bug in dmd. ldc 1.3 doesn't complain
in this case.
Arek
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/16/17 8:58 AM, Steven Schveighoffer wrote: However, I have found a better way to call postblit that involves the qualifiers than the way Variant currently does it. I'm going to submit a PR to fix these issues. https://github.com/dlang/phobos/pull/5694 -Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/16/17 9:04 AM, Kagamin wrote:
On Wednesday, 16 August 2017 at 12:58:25 UTC, Steven Schveighoffer wrote:
Use cases don't matter. What matters is: is it proper for Variant to
call the postblit (as it does currently) without regard for the
qualifiers?
Looks like it isn't, https://dpaste.dzfl.pl/183e6dae9867 - shared
reference counter with (probably) good enough postblit.
I have found that shared dtor doesn't work:
struct S
{
~this() shared {}
}
void main()
{
shared S s; // Error: shared method testshared.S.~this is not
callable using a non-shared object
}
But shared postblit does work.
Kind of a bummer. Though I think we need some work on the destructors
and postblits -- you can't overload them based on qualifiers.
But that isn't a concern for Variant. It is only calling the postblit,
which does work.
-Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Wednesday, 16 August 2017 at 12:58:25 UTC, Steven Schveighoffer wrote: Use cases don't matter. What matters is: is it proper for Variant to call the postblit (as it does currently) without regard for the qualifiers? Looks like it isn't, https://dpaste.dzfl.pl/183e6dae9867 - shared reference counter with (probably) good enough postblit.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/16/17 6:23 AM, Kagamin wrote: On Tuesday, 15 August 2017 at 15:19:02 UTC, Steven Schveighoffer wrote: However, I'm not sure about the postblit being called afterward. Does a postblit need to be marked shared in order to work for shared types? Ideally yes, but it's difficult to come up with a good shared postblit, send and receive is probably the only use case for it :) Use cases don't matter. What matters is: is it proper for Variant to call the postblit (as it does currently) without regard for the qualifiers? However, I have found a better way to call postblit that involves the qualifiers than the way Variant currently does it. I'm going to submit a PR to fix these issues. -Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Tuesday, 15 August 2017 at 15:19:02 UTC, Steven Schveighoffer wrote: However, I'm not sure about the postblit being called afterward. Does a postblit need to be marked shared in order to work for shared types? Ideally yes, but it's difficult to come up with a good shared postblit, send and receive is probably the only use case for it :)
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Tuesday, 15 August 2017 at 21:27:49 UTC, Arek wrote:
Yes, but this doesn't compile:
import std.stdio;
import std.concurrency;
struct A
{
int t;
int r;
int method() shared
{
return 0;
}
}
void consumer()
{
shared a = receiveOnly!(shared A)();
}
void main()
{
auto cons = spawn();
send(cons, shared A());
}
AIU you use struct as a Unique-like wrapper for shared class
object. Extract the object and send it, then wrap again on
arrival.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/15/17 5:27 PM, Arek wrote:
On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:
Well, no wrapper is actually needed here:
class A
{
int method() shared;
}
void consumer()
{
shared a = receiveOnly!(shared A)();
}
void producer()
{
auto cons = spawn();
send(cons, new shared A());
}
Yes, but this doesn't compile:
import std.stdio;
import std.concurrency;
struct A
{
int t;
int r;
int method() shared
{
return 0;
}
}
void consumer()
{
shared a = receiveOnly!(shared A)();
}
void main()
{
auto cons = spawn();
send(cons, shared A());
}
The issue is that send cannot handle shared value types due to a bug in
the implementation. In essence, there is no reason to send a shared A --
it's an unrelated copy and not actually shared.
You can send a shared reference to an A just fine:
static shared A a;
send(tid, ); // works
You *should* be able to send a shared(A). There is no reason to disallow
it. But again, it's not super useful.
This very simple code also doesn't compile:
shared struct S
{
int i;
~this()
{
}
}
void main()
{
shared s = shared S();
}
In general, shared structs with postblit and destructor make problems.
postblit should work.
It does look like destructors are a problem. It appears the compiler
attempts to call the destructor while unshared.
-Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:
Well, no wrapper is actually needed here:
class A
{
int method() shared;
}
void consumer()
{
shared a = receiveOnly!(shared A)();
}
void producer()
{
auto cons = spawn();
send(cons, new shared A());
}
Yes, but this doesn't compile:
import std.stdio;
import std.concurrency;
struct A
{
int t;
int r;
int method() shared
{
return 0;
}
}
void consumer()
{
shared a = receiveOnly!(shared A)();
}
void main()
{
auto cons = spawn();
send(cons, shared A());
}
This very simple code also doesn't compile:
shared struct S
{
int i;
~this()
{
}
}
void main()
{
shared s = shared S();
}
In general, shared structs with postblit and destructor make
problems.
Arek
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/15/17 10:42 AM, Kagamin wrote: https://github.com/dlang/phobos/blob/master/std/variant.d#L623 memcpy(, cast(const(void*)) , rhs.sizeof); should be ok to cast unconditionally Agreed, the T value being copied has already been copied onto the stack, so it's not really shared. However, I'm not sure about the postblit being called afterward. Does a postblit need to be marked shared in order to work for shared types? I think also I understand why it's working for shared int* but not shared int -- IFTI automatically infers tail-modified for such things to cut down on instantiations. tail-modified means the head is not shared (which is in this case more accurate). This means the cast is not necessary, since you are copying unqualified data. -Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
https://github.com/dlang/phobos/blob/master/std/variant.d#L623 memcpy(, cast(const(void*)) , rhs.sizeof); should be ok to cast unconditionally
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/14/17 5:27 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via Digitalmars-d-
learn wrote:
On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn
wrote:
I have the folowing problem:
I like to envelope the class object in struct to control the
destruction moment and then send this object to another
thread/fiber (or task, cause I use vibe-d).
I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared
because of issues with Variant, which they use internally.
This can't be a correct statement. This is the whole point of shared.
What's incorrect about it? It's a longstanding issue that because Variant
can't contain shared data, send and receive do not work with shared.
The implementation details aren't important. From the documentation (and
no, this documentation is not wrong):
From spawn:
args must not have unshared aliasing. In other words, all arguments to
fn must either be shared or immutable or have no pointer indirection.
This is necessary for enforcing isolation among threads.
From send:
As with std.concurrency.spawn, T must not have unshared aliasing.
So clearly passing shared pointers or things containing shared pointers
should work fine.
As I was building code to test, I found that it does actually work for
shared int pointers:
import std.concurrency;
import std.typecons;
import std.stdio;
import core.thread;
void threadfunc()
{
bool done = false;
while(!done)
{ receive(
(shared(int)*foo) {*foo = 5; done = true;},
(Variant v) {}
);
}
}
shared int f;
void main()
{
auto tid = spawn();
tid.send();
Thread.sleep(1.seconds);
writeln(f);
}
No error, completes as expected, and outputs 5.
So it looks like this is really a straight up bug and has nothing to do
with the shared type qualifier. It is documented as working, and does
work in some cases.
I think if the shared item is not a reference, it is doing something
different, and this is incompatible with something in the
implementation. Indeed, if you attempt to send unshared references, you
get an assert. However, if you send a shared int (not a pointer), you
get a bunch of compiler errors. Clearly the implementation expects it to
work, as it doesn't fail the logical checks.
-Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
Well, no wrapper is actually needed here:
class A
{
int method() shared;
}
void consumer()
{
shared a = receiveOnly!(shared A)();
}
void producer()
{
auto cons = spawn();
send(cons, new shared A());
}
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 22:22:58 UTC, Arek wrote:
I've found some simple workaround for this problem:
import std.stdio;
import std.concurrency;
struct Envelope(T) if (is(T == class)) // for simplicity of
this example, only classes
{
shared(T)[] obj;
this(shared T o)
{
this.obj = [o];
}
T get() @property nothrow @nogc
{
return cast() obj[0];
}
}
class A
{
}
void consumer()
{
auto r = receiveOnly!(Envelope!(A))();
writeln("Got: ", typeof(r).stringof);
}
void main()
{
auto cons = spawn();
auto o = Envelope!A(new A());
send(cons, o);
}
Shared object can be encapsulated in the array. In case of
other (non-class) types the pointer can be used, and get()
should return ref to the pointed object (after stripping off
the shared qualifier).
Rather like this:
struct Sendable(T)
{
shared T o;
alias o this;
}
import std.concurrency;
class A
{
int method() shared;
}
void consumer()
{
shared A a = receiveOnly!(Sendable!(A))();
}
void producer()
{
auto cons = spawn();
shared A a = new shared A();
send(cons, Sendable!A(a));
}
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 20:13:28 UTC, Arek wrote: If I can ensure the uniqueness of the object, there is no need to "share" it or synchronize the access. You use manually managed multithreading, that's why you need shared. And because compiler can't verify uniqueness, you are requested to do it manually by casting.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 21:27:48 UTC, Jonathan M Davis wrote:
On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via
Digitalmars-d- learn wrote:
On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn
wrote:
> On Saturday, August 12, 2017 18:57:44 Arek via
> Digitalmars-d-learn
wrote:
>> I have the folowing problem:
>> I like to envelope the class object in struct to control the
>> destruction moment and then send this object to another
>> thread/fiber (or task, cause I use vibe-d).
>>
>> I can't find any method to make it working. Any ideas?
>
> Unfortunately, send and receive do not currently work with
> shared because of issues with Variant, which they use
> internally.
This can't be a correct statement. This is the whole point of
shared.
What's incorrect about it? It's a longstanding issue that
because Variant can't contain shared data, send and receive do
not work with shared. You can send and receive mutable data
with no indirections, and AFAIK, you can send immutable data
(though the OP is apparently having problems with that, so I
guess that that doesn't work completely, though I definitely
recall doing so previously), but you get compilation errors if
you try to send shared data. For instance, this code
import std.concurrency;
void main()
{
static void func(Tid parent)
{
auto received = receiveOnly!(shared int[string]);
}
shared int[string] aa;
auto tid = spawn(, thisTid);
send(tid, aa);
}
I've found some simple workaround for this problem:
import std.stdio;
import std.concurrency;
struct Envelope(T) if (is(T == class)) // for simplicity of this
example, only classes
{
shared(T)[] obj;
this(shared T o)
{
this.obj = [o];
}
T get() @property nothrow @nogc
{
return cast() obj[0];
}
}
class A
{
}
void consumer()
{
auto r = receiveOnly!(Envelope!(A))();
writeln("Got: ", typeof(r).stringof);
}
void main()
{
auto cons = spawn();
auto o = Envelope!A(new A());
send(cons, o);
}
Shared object can be encapsulated in the array. In case of other
(non-class) types the pointer can be used, and get() should
return ref to the pointed object (after stripping off the shared
qualifier).
send() could encapsulate itself shared objectes.
Arek
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via Digitalmars-d-
learn wrote:
> On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
> > On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn
wrote:
> >> I have the folowing problem:
> >> I like to envelope the class object in struct to control the
> >> destruction moment and then send this object to another
> >> thread/fiber (or task, cause I use vibe-d).
> >>
> >> I can't find any method to make it working. Any ideas?
> >
> > Unfortunately, send and receive do not currently work with shared
> > because of issues with Variant, which they use internally.
>
> This can't be a correct statement. This is the whole point of shared.
What's incorrect about it? It's a longstanding issue that because Variant
can't contain shared data, send and receive do not work with shared. You can
send and receive mutable data with no indirections, and AFAIK, you can send
immutable data (though the OP is apparently having problems with that, so I
guess that that doesn't work completely, though I definitely recall doing so
previously), but you get compilation errors if you try to send shared data.
For instance, this code
import std.concurrency;
void main()
{
static void func(Tid parent)
{
auto received = receiveOnly!(shared int[string]);
}
shared int[string] aa;
auto tid = spawn(, thisTid);
send(tid, aa);
}
will fail to compile with
/usr/local/include/dmd/std/variant.d(625): Error: function
core.stdc.string.memcpy (return scope void* s1, scope const(void*) s2, ulong
n) is not callable using argument types (ubyte[32]*, shared(int[string])*,
ulong)
/usr/local/include/dmd/std/variant.d(625): Error: function
core.stdc.string.memcpy (return scope void* s1, scope const(void*) s2, ulong
n) is not callable using argument types (ubyte[32]*, shared(int)*, ulong)
/usr/local/include/dmd/std/variant.d(424): Error: template instance
std.variant.VariantN!32LU.VariantN.opAssign!(shared(int)) error
instantiating
/usr/local/include/dmd/std/variant.d(645):instantiated from here:
handler!(shared(int[string]))
/usr/local/include/dmd/std/variant.d(559):instantiated from here:
opAssign!(shared(int[string]))
/usr/local/include/dmd/std/concurrency.d(101):instantiated from
here: __ctor!(shared(int[string]))
/usr/local/include/dmd/std/concurrency.d(606):... (2 instantiations,
-v to show) ...
/usr/local/include/dmd/std/concurrency.d(576):instantiated from
here: _send!(shared(int[string]))
q.d(13):instantiated from here: send!(shared(int[string]))
whereas if you use something like int, it has no problem. It won't even
compile if you give it a shared int, and that arguably should just be
converting to and from int, since shared int has no indirections.
There are at least two bugs reported on the issue:
https://issues.dlang.org/show_bug.cgi?id=13262
https://issues.dlang.org/show_bug.cgi?id=14893
I suspect that the solution involves making it so that std.concurrency uses
something other than Variant to store the data being sent, but regardless,
no one has fixed the issue yet. So, while send and receive work fine for
simple message, they're fairly crippled when it comes to complex data and
have been for their entire existence AFAIK.
- Jonathan M Davis
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 19:22:23 UTC, Steven Schveighoffer wrote: On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote: On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote: I have the folowing problem: I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d). I can't find any method to make it working. Any ideas? Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally. This can't be a correct statement. This is the whole point of shared. -Steve First of all, I'm not native English speaker, so forgive me possible misunderstanding In my opinion the whole problem of 'shared' is that when I use send/receive operation (I believe it's should be something similar to go's channels or erlang messages) I do not want to share the memory so any "shared" concept is useless. What I really would like to get is possibility to make: // in first thread Unique!MyObject msg = new MyObject() send(msg.release); // in other thread: auto msg = receiveOnly!(Unique!MyObject)(); My object disappears in "send" operation and its new instance is constructed on receive. If I can ensure the uniqueness of the object, there is no need to "share" it or synchronize the access. But now such operation is impossible and the compiler forces me to cope with shared if MyObject has any references to other areas of memory. Anyway, thanks for all replies. Arek
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:
On Sunday, August 13, 2017 16:40:03 crimaniak via
Digitalmars-d-learn wrote:
More of this, I think, you can't avoid __gshared for any
complex work. Even mutexes from Phobos doesn't support shared,
so I had to 'cowboy with __gshared' when implementing my site
engine.
The way to handle shared is to protect the section of code
that's using the shared object with either a mutex or
synchronized block, and then you cast away shared from the
object within that section and operate on it as thread-local.
When you're done, you make sure that you don't have any
thread-local references to the data, and you release the mutex
or exit the synchronized block. e.g. something like
shared T sharedObj = getSharedObj();
synchronized(mutex)
{
T nonSharedObj = cast(T)sharedObject
// do stuff...
// make sure that no references to nonSharedObj have escaped
}
// now, there's just the shared version of the object
Yeah, and this is what i'm doing now (more or less).
To be more precise, I don't even want to synchronize access to
the shared resource between the threads. I just want to move the
object from one thread to another.
Of course I could copy the local object, but my obcjecs has
indirect references to others, forming kind of tree.
I like the idea of channels in Go. I've tried to get something
similiar with send/receive.
In this case, object could be also immutable, because once there
are created (in the deserialization process) they will no be
modified. I just have to emit them into another task (to be
honest, I use fiber, so it's not even another thread, and they
will not be accessed in parallel).
But all this language protections makes the issue unexpectedly
complicated.
And no, this isn't ideal, but the only semi-decent solution
that's been proposed that safely casts away shared for you is
synchronized classes, which Andrei describes in TDPL but have
never been implemented. And because they can only safely strip
off the outermost layer of shared, they're of questionable
usefulness anyway. Ultimately, even with synchronized classes,
in many situations, the programmer is going to have to
carefully cast away shared to operate on the object within a
protected context.
Now, the fact that the mutex objects don't handle shared
correctly is another issue entirely. Having to cast away shared
from mutexes is dumb, because you're obviously not going to be
protecting them with a mutex, and their operations have to be
atomic anyway for them to do what they do. So, that definitely
needs to be fixed. However, I believe that it _has_ been fixed
in master, and it might have made it into a release now, but
I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable
as shared like it should be.
In general though, the idea is that you simply don't operate on
shared objects except via atomic operations. Otherwise, you
risk concurrency problems. And really, this is the same as what
you'd do in C/C++, except that in C/C++, it doesn't catch you
when you operate on an object that's shared across threads with
non-atomic operations (because the object isn't explicitly
typed as shared), and you don't have to cast away shared to do
non-atomic operations. So, having to cast away shared is the
price of getting the protection against accidentally using
non-atomic operations on a shared object as well as the price
we pay to be able to have the type system distinguish between
shared and thread-local objects so that it's able to optimize
based on the knowledge that an object is thread-local.
Ultimately though, you're doing the same thing that you'd do in
C++ if you're handling concurrency safely. You just have to
explicitly mark stuff as shared and carefully cast away shared
in certain, protected contexts.
Using __gshared in extern(D) code is just asking for it,
because then you have an object that the compiler thinks is
thread-local but isn't, and you risk subtle and nasty bugs as a
result. __gshared is only intended for binding to extern(C),
global variables. To an extent, you can get away with using it
with extern(D) variables, but that's not its intended purpose,
and you risk running afoul of the compiler and what it chooses
to do based on the assumption that the object is thread-local.
- Jonathan M Davis
Thanks for the explanation! It would be good to have a
comprehensive article on this subject.
Arek
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote: On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote: I have the folowing problem: I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d). I can't find any method to make it working. Any ideas? Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally. This can't be a correct statement. This is the whole point of shared. -Steve
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 03:40:26 UTC, Jonathan M Davis wrote: On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote: I have the folowing problem: I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d). I can't find any method to make it working. Any ideas? Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally. - Jonathan M Davis That's what I suspected. Thanks for the confirmation. I was not sure if I was doing everything right. Arek
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote: [snip] Now, the fact that the mutex objects don't handle shared correctly is another issue entirely. Having to cast away shared from mutexes is dumb, because you're obviously not going to be protecting them with a mutex, and their operations have to be atomic anyway for them to do what they do. So, that definitely needs to be fixed. However, I believe that it _has_ been fixed in master, and it might have made it into a release now, but I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable as shared like it should be. My fixes for shared(Mutex) - https://github.com/dlang/druntime/pull/1728 - are part of the DMD v2.074.0 release, and should propagate to LDC with their 1.4 release respectively. GDC master now at druntime/phobos 2.074.1 - https://github.com/D-Programming-GDC/GDC/pull/539 and they're in the process of moving to 2.075.0 - https://github.com/D-Programming-GDC/GDC/pull/542. Unfortunately, I forgot to write a changelog entry when I was working on the PR and later didn't have time to do so, before the 2.074.0 release. So shared (Mutex) should be in good shape now. I should supplement the ddoc unittest to address the transitive you issues you talked about. Currently it only shows synchronized use of a plain value type without indirections, which is uninteresting. The example I'm thinking about is casting with HeadUnsharedOf / TailSharedOf a pointer to a singly-linked list, because with pointers you could show the shared(Node*) -> shared(Node)* transition in the type system, which unfortunately can't be demonstrated with implicitly reference types (classes). Speaking of TailSharedOf, Jonathan could take a look at this PR: https://github.com/dlang/druntime/pull/1605? I and Andrei already approved it, but I'd like to get a third opinion since this is a breaking change and Martin seems be too busy the moment.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Sunday, August 13, 2017 16:40:03 crimaniak via Digitalmars-d-learn wrote:
> More of this, I think, you can't avoid __gshared for any complex
> work. Even mutexes from Phobos doesn't support shared, so I had
> to 'cowboy with __gshared' when implementing my site engine.
The way to handle shared is to protect the section of code that's using the
shared object with either a mutex or synchronized block, and then you cast
away shared from the object within that section and operate on it as
thread-local. When you're done, you make sure that you don't have any
thread-local references to the data, and you release the mutex or exit the
synchronized block. e.g. something like
shared T sharedObj = getSharedObj();
synchronized(mutex)
{
T nonSharedObj = cast(T)sharedObject
// do stuff...
// make sure that no references to nonSharedObj have escaped
}
// now, there's just the shared version of the object
And no, this isn't ideal, but the only semi-decent solution that's been
proposed that safely casts away shared for you is synchronized classes,
which Andrei describes in TDPL but have never been implemented. And because
they can only safely strip off the outermost layer of shared, they're of
questionable usefulness anyway. Ultimately, even with synchronized classes,
in many situations, the programmer is going to have to carefully cast away
shared to operate on the object within a protected context.
Now, the fact that the mutex objects don't handle shared correctly is
another issue entirely. Having to cast away shared from mutexes is dumb,
because you're obviously not going to be protecting them with a mutex, and
their operations have to be atomic anyway for them to do what they do. So,
that definitely needs to be fixed. However, I believe that it _has_ been
fixed in master, and it might have made it into a release now, but I'm not
sure. So, core.sync.mutex.Mutex _should_ now be useable as shared like it
should be.
In general though, the idea is that you simply don't operate on shared
objects except via atomic operations. Otherwise, you risk concurrency
problems. And really, this is the same as what you'd do in C/C++, except
that in C/C++, it doesn't catch you when you operate on an object that's
shared across threads with non-atomic operations (because the object isn't
explicitly typed as shared), and you don't have to cast away shared to do
non-atomic operations. So, having to cast away shared is the price of
getting the protection against accidentally using non-atomic operations on a
shared object as well as the price we pay to be able to have the type system
distinguish between shared and thread-local objects so that it's able to
optimize based on the knowledge that an object is thread-local. Ultimately
though, you're doing the same thing that you'd do in C++ if you're handling
concurrency safely. You just have to explicitly mark stuff as shared and
carefully cast away shared in certain, protected contexts.
Using __gshared in extern(D) code is just asking for it, because then you
have an object that the compiler thinks is thread-local but isn't, and you
risk subtle and nasty bugs as a result. __gshared is only intended for
binding to extern(C), global variables. To an extent, you can get away with
using it with extern(D) variables, but that's not its intended purpose, and
you risk running afoul of the compiler and what it chooses to do based on
the assumption that the object is thread-local.
- Jonathan M Davis
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote: > I have the folowing problem: > I like to envelope the class object in struct to control the > destruction moment and then send this object to another > thread/fiber (or task, cause I use vibe-d). > > I can't find any method to make it working. Any ideas? Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally. - Jonathan M Davis
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Sunday, 13 August 2017 at 11:35:05 UTC, Arek wrote: Yeah, I've read this. But conurrency.send cannot pass immutable object. The same story with Unique. Sorry, read this as 'efficient immutable'. More over, "shared" looks rather like unfinished concept. Yes, exactly. Anyway, _gshared looks very promising, so I will try to work out any approach. More of this, I think, you can't avoid __gshared for any complex work. Even mutexes from Phobos doesn't support shared, so I had to 'cowboy with __gshared' when implementing my site engine.
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Sunday, 13 August 2017 at 02:50:13 UTC, crimaniak wrote: On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote: I have the folowing problem: I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d). I can't find any method to make it working. Any ideas? I tried it too some time ago. Then I read Alexandrescu book and realized that the authors of the language do not want anyone to do this. Long story short, just plan your application so that each complex object is monitored by only one thread/task, and pass not objects, but messages (immutable structs) about what to do with them. Yeah, I've read this. But conurrency.send cannot pass immutable object. The same story with Unique. What means, that if I have complex objects which I want to transfer between threads I must keep them somewhere in global memory and share among threads. I can use send/receive only to signal availability of their existance. What I need is kind of "move" operation beetwen threads. More over, "shared" looks rather like unfinished concept. It is really difficult to create proper struct (with postblit and destructor) working as shared object. I even have no clue what is exact semantics of "shared" types. Language specification is a little laconic. Anyway, _gshared looks very promising, so I will try to work out any approach. Thanks for answer. Arek
Re: Does anyone understand how to use "shared" types with concurrency send/receive functions?
On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote: I have the folowing problem: I like to envelope the class object in struct to control the destruction moment and then send this object to another thread/fiber (or task, cause I use vibe-d). I can't find any method to make it working. Any ideas? I tried it too some time ago. Then I read Alexandrescu book and realized that the authors of the language do not want anyone to do this. Long story short, just plan your application so that each complex object is monitored by only one thread/task, and pass not objects, but messages (immutable structs) about what to do with them.
