Re: Sending an immutable object to a thread
On 07/23/2015 06:48 AM, Frank Pagliughi wrote:
So, passing a pointer to a stack-based reference from one thread is
another is not necessarily a good thing to do, as the original reference
might disappear while the thread is using it.
Right.
Is there a way to get the address of the actual heap object from a class
reference?
It is possible by casting the reference to a pointer type:
import std.stdio;
class B
{
int i;
this(int i) { this.i = i; }
}
class D : B
{
int j;
this (int j) {
super(j);
this.j = j + 1;
}
}
void main()
{
auto r = new D(42);
writefln(Address of reference: %s, r);
writefln(Address of object : %s, cast(void*)r);
writefln(Address of i: %s, r.i);
writefln(Address of j: %s, r.j);
auto p = cast(void*)r;
auto r2 = cast(D)p;
writefln(i through another reference: %s, r2.i);
writefln(j through another reference: %s, r2.j);
}
Although the example casts to void*, ubyte* and others are possible as
well, and casting back to the correct class type seems to work:
Address of reference: 7FFFCB137580
Address of object : 7FFCB9407520
Address of i: 7FFCB9407530
Address of j: 7FFCB9407534
i through another reference: 42
j through another reference: 43
Ali
Re: Sending an immutable object to a thread
On Friday, 24 July 2015 at 18:55:26 UTC, Frank Pagliughi wrote:
So then, of course, I hope/wonder/assume that the pointer to
the heap is sufficient to keep the heap memory alive, and that
this would be OK from the GC perspective to do something like
this:
B* make_b_thing(int i) { cast(B*) new B(i); }
(You missed a return there.)
I haven't followed the discussion, so I may be missing the point
here. But if you're doing this so that the GC is aware of the
`new`ed B, then you really don't need to. The GC has no problems
with class types. Class references do keep objects alive.
That is, the above will not keep the created object any more
alive than the following:
B make_b_thing(int i) { return new B(i); }
That seems to work, but I guess I should try to force the
garbage collector to run to see if I can crash the program.
***BUT***: The really, really weird thing is that even though
you *think* that you have a pointer to a B object, you don't
really. Dereferencing is accepted by the compiler, but it plays
a nasty trick on you:
A B* is not a pointer to the memory of the object. It's a pointer
to a class reference. The class reference itself, B, is a pointer
to the memory of the object, under the hood.
Casting a B to B* makes as little sense as casting it to float*
or char*.
B* p = make_b_thing(42);
writefln(Address of pointer: %s, p);
writefln(Value of i: %s, p.i);
p really is a pointer to the memory of the object. But by typing
it B* you're stating that it's a pointer to a class reference,
which is wrong.
p.i does two dereferences where one would be correct. First, p is
dereferenced. The resulting data is really that of the object.
But it's typed B, so the compiler thinks it's a class reference.
So it takes first bytes of the actual object data, interprets it
as a pointer, dereferences it, and assumes to see the object data
there (wherever that is). Getting the i field of that garbage
location results in some garbage data, of course.
writefln(Value of i: %s, (*p).i);
Same as above. When p is a pointer, then p.i becomes (*p).i
automatically in D.
writefln(Value of i: %s, (cast(B)p).i);
Here the type has only one level of indirection, as it should be.
And everything's fine.
This compiles and runs fine, but produces:
Address of pointer: 7F7EE77CF020
Value of i: 4445040
Value of i: 4445040
Value of i: 42
Maybe it's my C++ background talking, but that seems a bit
counter-intuitive.
I'd say it's your C++ background talking.
Re: Sending an immutable object to a thread
On Friday, 24 July 2015 at 18:02:58 UTC, Ali Çehreli wrote:
Although the example casts to void*, ubyte* and others are
possible as well, and casting back to the correct class type
seems to work:
Thanks, Ali.
I just tried a few things, and apparently, you don't need to go
to a different type or void. You can make it a pointer the the
actual type, which might be good for self-documentation or
pattern matching. But what I find somewhat odd and fascinating is
what you show, that for the reference, r:
cast(D*)r != r
So this code all works out:
import std.stdio;
class B
{
int i;
this(int i) { this.i = i; }
}
void main()
{
auto r1 = new B(42),
r2 = r1;
writefln(Address of reference r1: %s, r1);
writefln(Address of reference r2: %s, r2);
writefln(Address of object r1 : %s, cast(B*)r1);
writefln(Address of object r2 : %s, cast(B*)r2);
assert(cast(B*)r1 == cast(B*)r2);
assert(cast(B*)r1 != r1);
assert(cast(B*)r2 != r2);
}
and prints:
Address of reference r1: 7FFF1D4E4C70
Address of reference r2: 7FFF1D4E4C78
Address of object r1 : 7F01CD506000
Address of object r2 : 7F01CD506000
So then, of course, I hope/wonder/assume that the pointer to the
heap is sufficient to keep the heap memory alive, and that this
would be OK from the GC perspective to do something like this:
B* make_b_thing(int i) { cast(B*) new B(i); }
That seems to work, but I guess I should try to force the garbage
collector to run to see if I can crash the program.
***BUT***: The really, really weird thing is that even though you
*think* that you have a pointer to a B object, you don't really.
Dereferencing is accepted by the compiler, but it plays a nasty
trick on you:
B* p = make_b_thing(42);
writefln(Address of pointer: %s, p);
writefln(Value of i: %s, p.i);
writefln(Value of i: %s, (*p).i);
writefln(Value of i: %s, (cast(B)p).i);
This compiles and runs fine, but produces:
Address of pointer: 7F7EE77CF020
Value of i: 4445040
Value of i: 4445040
Value of i: 42
Maybe it's my C++ background talking, but that seems a bit
counter-intuitive.
Re: Sending an immutable object to a thread
On Friday, 24 July 2015 at 19:28:35 UTC, anonymous wrote: I haven't followed the discussion, so I may be missing the point here. I started by asking how to send a reference to an immutable class object from one thread to another if the reference is one of several parameters being sent. The concurrency library can't make a tuple for send/receive if the reference is immutable. The short answer is (probably) to use a rebindable reference, but a suggestion arose about the possible use of a pointer instead. So we've devolved into a discussion of how pointers to class objects work, and how to keep the heap memory alive as the pointer is sent to the second thread as the original reference goes out of scope in the first. A B* is not a pointer to the memory of the object. It's a pointer to a class reference. The class reference itself, B, is a pointer to the memory of the object, under the hood. Hahaha. My forehead is red from the number of times I've thought I've gotten it in this discussion. So I think I understand that when you start to peek under the hood, the language treats the reference and the heap memory as distinct entities, and that they work differently for struct's and classes. So then: is there a pointer notation to which you can cast the B reference, which thus points to the heap, but retains type identity of the heap object? And the reason I ask if I wanted to declare a type which is a mutable pointer to an immutable object that is on the GC heap, keeping type info? Or is the answer, no, just use Rebindable when necessary? At this point, I'm not asking what should I do, but what could I do.
Re: Sending an immutable object to a thread
On Friday, 24 July 2015 at 21:51:44 UTC, Frank Pagliughi wrote: So then: is there a pointer notation to which you can cast the B reference, which thus points to the heap, but retains type identity of the heap object? There's no straight forward way to do that. D has no types for the actual objects of classes. You'd have to generate a new struct type from B that you can point at. And the reason I ask if I wanted to declare a type which is a mutable pointer to an immutable object that is on the GC heap, keeping type info? Or is the answer, no, just use Rebindable when necessary? We only have Rebindable because there is no nice way in the language to do it. So, just use Rebindable, yes. You could look at what Rebindable does, and then do that, of course. But be sure that you understand all the nasty details.
Re: Sending an immutable object to a thread
On Thursday, 23 July 2015 at 09:05:12 UTC, Marc Schütz wrote:
It is not safe, but for a different reason: `mt` is already a
_reference_ to the actual object (that's how classes behave in
D). This reference is located in a register or on the stack,
and `mt` is therefore a pointer into the stack.
It's illegal to return that pointer from the function, because
it will become invalid once the function is left. Fortunately,
the compiler can detect simple cases like this one, and will
refuse to compile it:
Object* foo() {
Object o;
return o;
}
xx.d(3): Error: escaping reference to local o
Very interesting. You see, I am trying to resolve the distinction
be a value type and a reference type in D.
If Object were declared as a struct, this would make sense to
me. The object would be created on the stack as a temporary, and
it would disappear when the function exited. So returning a
pointer to it would be a very, very bad thing.
But a class object is allocated in the GC heap. I would have
guessed that, since a class reference is just essentially a
hidden pointer, that the address-of operator '' for a class
object would return the address into the heap... not the address
of the reference itself! Just a little syntactic sugar.
But that's not the case. I thought this was true:
class MyThing { ... };
MyThing a = new MyThing,
b = a;
assert(a == b); // Fails
In a weird way, that makes total sense to me, and no sense at all.
So, passing a pointer to a stack-based reference from one thread
is another is not necessarily a good thing to do, as the original
reference might disappear while the thread is using it.
Is there a way to get the address of the actual heap object from
a class reference? Or am I drifting to far into the realm of
unsafe.
This all goes back to my original question of passing an
immutable object from one thread to another. It is simple with
arrays, since there is a clear distinction between the array
reference and its contents. You can easily create a mutable
reference to immutable contents with an array.
But it seems rather convoluted with class objects.
So, in the end, it seems best to send a rebindable reference to
the other thread, and perhaps hide the mild ugliness behind a
library API that takes an immutable object and then sends the
rebindable version, like:
void send_message(Tid tid, immutable(Message) msg) {
send(tid, thisTid(), rebindable(msg));
}
That seems easy enough.
Thanks much for all the help.
Re: Sending an immutable object to a thread
On Wednesday, 22 July 2015 at 17:17:17 UTC, Frank Pagliughi wrote:
Or, to put it another way, getting threads out of the equation,
is this safe?
class MyThing { ... }
MyThing* create_a_thing() {
MyThing mt = new MyThing();
do_something_with(mt);
return mt;
}
void main() {
MyThing* pmt = create_a_thing();
// ...
}
The thing will remain alive for the duration of main() ??
It is not safe, but for a different reason: `mt` is already a
_reference_ to the actual object (that's how classes behave in
D). This reference is located in a register or on the stack, and
`mt` is therefore a pointer into the stack.
It's illegal to return that pointer from the function, because it
will become invalid once the function is left. Fortunately, the
compiler can detect simple cases like this one, and will refuse
to compile it:
Object* foo() {
Object o;
return o;
}
xx.d(3): Error: escaping reference to local o
Re: Sending an immutable object to a thread
On Wednesday, 22 July 2015 at 09:04:49 UTC, Marc Schütz wrote:
But as long as the original pointer is still on the stack, that
one _will_ keep the object alive. It is only a problem if all
pointers to a GC managed object are stored in places the GC
isn't informed about.
Sorry, I have gotten confused. In Ali's example, the pointer to a
class object (via the address-of '' operator) actually points
into the GC heap. It is *not* a pointer to a pointer, right?
My reading of the Garbage web doc page is that this pointer to
memory in the GC heap is sufficient (by some magic) to keep the
memory alive, in and of itself.
So the pointer, passed to the other thread is sufficient to keep
the memory alive, even if the original reference disappears.
Or, to put it another way, getting threads out of the equation,
is this safe?
class MyThing { ... }
MyThing* create_a_thing() {
MyThing mt = new MyThing();
do_something_with(mt);
return mt;
}
void main() {
MyThing* pmt = create_a_thing();
// ...
}
The thing will remain alive for the duration of main() ??
Thanks
Re: Sending an immutable object to a thread
On Wednesday, 22 July 2015 at 17:17:17 UTC, Frank Pagliughi wrote:
On Wednesday, 22 July 2015 at 09:04:49 UTC, Marc Schütz wrote:
But as long as the original pointer is still on the stack,
that one _will_ keep the object alive. It is only a problem if
all pointers to a GC managed object are stored in places the
GC isn't informed about.
Sorry, I have gotten confused. In Ali's example, the pointer to
a class object (via the address-of '' operator) actually
points into the GC heap. It is *not* a pointer to a pointer,
right?
My reading of the Garbage web doc page is that this pointer to
memory in the GC heap is sufficient (by some magic) to keep the
memory alive, in and of itself.
So the pointer, passed to the other thread is sufficient to
keep the memory alive, even if the original reference
disappears.
Or, to put it another way, getting threads out of the equation,
is this safe?
class MyThing { ... }
MyThing* create_a_thing() {
MyThing mt = new MyThing();
do_something_with(mt);
return mt;
}
void main() {
MyThing* pmt = create_a_thing();
// ...
}
The thing will remain alive for the duration of main() ??
Thanks
No.
this is actually returning an address of a temporary.
Re: Sending an immutable object to a thread
On Tuesday, 21 July 2015 at 21:50:35 UTC, rsw0x wrote: On Tuesday, 21 July 2015 at 21:44:07 UTC, rsw0x wrote: On Sunday, 19 July 2015 at 17:12:07 UTC, rsw0x wrote: [...] wow, I don't even remember posting this. This is (mostly) wrong, but I'm unsure if a pointer to another pointer on the stack would correctly keep its object alive(but, I believe this would just be a bug I think,) If the pointer was pointing to a pointer on the heap, then AFAICT it would keep it alive. addendum: http://dlang.org/garbage.html Pointers in D can be broadly divided into two categories: Those that point to garbage collected memory, and those that do not. Examples of the latter are pointers created by calls to C's malloc(), pointers received from C library routines, pointers to static data, pointers to objects on the stack, etc. and those that do not ... pointers to objects on the stack, etc. I believe this implies that it would *not* keep the object alive. Sorry for the confusion/noise. But as long as the original pointer is still on the stack, that one _will_ keep the object alive. It is only a problem if all pointers to a GC managed object are stored in places the GC isn't informed about.
Re: Sending an immutable object to a thread
On Wednesday, 22 July 2015 at 09:04:49 UTC, Marc Schütz wrote: On Tuesday, 21 July 2015 at 21:50:35 UTC, rsw0x wrote: On Tuesday, 21 July 2015 at 21:44:07 UTC, rsw0x wrote: [...] addendum: http://dlang.org/garbage.html [...] [...] I believe this implies that it would *not* keep the object alive. Sorry for the confusion/noise. But as long as the original pointer is still on the stack, that one _will_ keep the object alive. It is only a problem if all pointers to a GC managed object are stored in places the GC isn't informed about. correct, I managed to confuse myself :o)
Re: Sending an immutable object to a thread
On Sunday, 19 July 2015 at 17:12:07 UTC, rsw0x wrote: On Sunday, 19 July 2015 at 17:04:07 UTC, Frank Pagliughi wrote: [...] Oh, yes, pointer. Ha! I didn't even think of that. Thanks. I'm not familiar with how garbage collection works in D. If the initial reference goes out of scope, and you just have a pointer - in another thread, no less - then are you still guaranteed that the object will not disappear while the pointer exists? [...] a pointer to a pointer(or in this case, a reference) does not keep it alive. wow, I don't even remember posting this. This is (mostly) wrong, but I'm unsure if a pointer to another pointer on the stack would correctly keep its object alive(but, I believe this would just be a bug I think,) If the pointer was pointing to a pointer on the heap, then AFAICT it would keep it alive.
Re: Sending an immutable object to a thread
On Tuesday, 21 July 2015 at 21:44:07 UTC, rsw0x wrote: On Sunday, 19 July 2015 at 17:12:07 UTC, rsw0x wrote: [...] wow, I don't even remember posting this. This is (mostly) wrong, but I'm unsure if a pointer to another pointer on the stack would correctly keep its object alive(but, I believe this would just be a bug I think,) If the pointer was pointing to a pointer on the heap, then AFAICT it would keep it alive. addendum: http://dlang.org/garbage.html Pointers in D can be broadly divided into two categories: Those that point to garbage collected memory, and those that do not. Examples of the latter are pointers created by calls to C's malloc(), pointers received from C library routines, pointers to static data, pointers to objects on the stack, etc. and those that do not ... pointers to objects on the stack, etc. I believe this implies that it would *not* keep the object alive. Sorry for the confusion/noise.
Re: Sending an immutable object to a thread
On Sunday, 19 July 2015 at 17:12:07 UTC, rsw0x wrote:
a pointer to a pointer(or in this case, a reference) does not
keep it alive.
Interesting. If you de-reference the pointer and assign it back,
do you get back the keep-alive? Like, in the receiving thread:
void threadFunc()
{
receive((Tid cli, immutable(Message) *m) {
immutable(Message) msg = *m; // ---
int retCode = do_something_with(msg);
send(cli, retCode);
});
}
I assume that even if so, there is a race condition there. You
would need to keep the original reference alive until at least
the msg = *m assignment happened, right?
Or... could you tell the GC to leave the memory alone until the
thread gets it? Like in the sending thread:
Tid tid = spawn(threadFunc);
auto p = cast(void*) msg;
GC.addRoot(p);
GC.setAttr(p, GC.BlkAttr.NO_MOVE);
send(tid, thisTid(), msg);
//...
Is that possible? Is it efficient enough to do if you're sending
lots and lots of messages?
Thanks.
Re: Sending an immutable object to a thread
It is a pitty that although Variant is the default message type in
concurrency, it still has issues:
https://issues.dlang.org/buglist.cgi?quicksearch=variant%20concurrencylist_id=202195
It looks like passing a pointer to an immutable(Message) works as well:
import std.stdio;
import std.concurrency;
class Message
{
int i;
this(int i) immutable
{
this.i = i;
}
}
int do_something_with(immutable(Message) msg)
{
writefln(msg.i is %s, msg.i);
return 0;
}
void threadFunc()
{
receive((Tid cli, immutable(Message) *msg) {
int retCode = do_something_with(*msg);
send(cli, retCode);
});
ownerTid.send(42);
}
void main()
{
auto msg = new immutable(Message)(100);
Tid tid = spawn(threadFunc);
send(tid, thisTid(), msg);// -- POINTER
receiveOnly!int();
}
Ali
Re: Sending an immutable object to a thread
On Sunday, 19 July 2015 at 17:04:07 UTC, Frank Pagliughi wrote: [...] Oh, yes, pointer. Ha! I didn't even think of that. Thanks. I'm not familiar with how garbage collection works in D. If the initial reference goes out of scope, and you just have a pointer - in another thread, no less - then are you still guaranteed that the object will not disappear while the pointer exists? [...] a pointer to a pointer(or in this case, a reference) does not keep it alive.
Re: Sending an immutable object to a thread
It looks like passing a pointer to an immutable(Message) works
as well:
Oh, yes, pointer. Ha! I didn't even think of that. Thanks.
I'm not familiar with how garbage collection works in D. If the
initial reference goes out of scope, and you just have a pointer
- in another thread, no less - then are you still guaranteed that
the object will not disappear while the pointer exists?
Like if I did something akin to:
// ...like before...
void send_msg(Tid tid, int n)
{
auto msg = new immutable(Message)(n);
send(tid, thisTid(), msg);
}
void main()
{
Tid tid = spawn(threadFunc);
send_msg(tid, 100);
receiveOnly!int();
}
Do I know that the message object won't be garbage collected
before the thread finishes with it? (I realize this is a very
artificial example, but something like this could happen in a
bigger library).
Frank
Re: Sending an immutable object to a thread
OK, I found a couple of solutions, though if anyone can tell me
something better, I would love to hear it.
By making an alias to a rebindable reference, the receive() was
able to create the tuple. So I renamed the class MessageType:
class MessageType { ... };
and then made a Message an immutable one of these:
alias immutable(MessageType) Message;
and finally made a VarMessage as a rebindable Message (thus, a
mutable reference to an immutable object):
alias Rebindable!(Message) VarMessage;
[I will likely rethink these names, but anyway... ]
Now I can send a reference to an immutable object across threads.
The receiver wants the VarMessage:
receive(
(Tid cli, VarMessage msg) {
int retVal = do_something_with(msg);
send(cli, retVal);
}
);
and a few different things work to send the object:
auto msg = new Message(...);
send(tid, thisTid(), VarMessage(msg));
or:
send(tid, thisTid(), rebindable(msg));
or:
VarMessage vmsg = new Message(...);
send(tid, thisTid(), vmsg);
A second way that seems plausible is to just make the message a
var type using a struct and then send a copy to the thread. This
seems viable since the vast bulk of the message is a string
payload, and thus the size of the struct is pretty small.
