Re: [go-nuts] CGO - Passing pointer to C

[Michael Jones]

Speaking for the Rick Hudson Fan Club, thank you for your wonderful,
amazing work on the Go GC...

https://blog.golang.org/ismmkeynote

...and for this reminder that we don't want programming practice to
preclude continuing innovations.

On Wed, Dec 4, 2019 at 9:52 AM Rick Hudson  wrote:

> Breaking the Go CGO pointer rules comes up periodically and the rules
> have not changed. Applications have lived with the rules simply
> because breaking them results in revisiting the application code
> every time a new Go release comes out. Did the compiler improve and
> some object is now allocated on the stack instead of the heap? Did the
> runtime borrow some MESH [1] virtual memory page fragmentation
> techniques but improve them for Go by updating pointers to reducing
> TLB pressure? Is there value in moving objects from NVRAM to DRAM
> and updating pointers? And so forth and so on. Nobody knows if any of
> this will ever happen but the Go CGO pointer rules leave open the
> possibility.
>
>
> [1] Bobby Powers, David Tench, Emery D. Berger, and Andrew McGregor. 2019.
> Mesh: Compacting Memory Management for C/C++ Applications. In
> Proceedings of the 40th ACM SIGPLAN Conference on Programming Language
> Design and Implementation (PLDI ’19), June 22ś26, 2019, Phoenix, AZ, USA.
> ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3314221.3314582
>
>
> On Wednesday, December 4, 2019 at 11:14:17 AM UTC-5, Ian Lance Taylor
> wrote:
>>
>> On Wed, Dec 4, 2019 at 6:48 AM Robert Johnstone 
>> wrote:
>> >
>> > Thanks for the quick reply.  I had not considered the write barriers,
>> but if the Go objects are "live" longer than the held in C, it would work.
>>
>> The write barriers do not look only at the pointer being stored, they
>> also look at the contents of memory being stored into.  That is why C
>> code must never store a Go pointer into Go memory.
>>
>>
>> > I definitely agree that there are risks associated with this approach.
>> We are giving up some of the safety of Go.  Unfortunately, we are using cgo
>> for more than some computation, so we have live objects in C as well, and
>> so we cannot completely escape the manual memory management required.
>> >
>> > What is the state of plans for a moving garbage collector?  This would
>> definitely wreck havoc if any pointers to Go memory were held in C.
>>
>> There are no current plans for a moving garbage collector.
>>
>> I cannot promise that no other changes will break this approach.
>> Obviously we won't consider bug reports for code that breaks the cgo
>> rules.
>>
>> Ian
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/23885bd0-4802-40c7-b4c9-52541e92029f%40googlegroups.com
> 
> .
>


-- 

*Michael T. jonesmichael.jo...@gmail.com *

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALoEmQwZiyh7wjL5OokVKoPkw-8tjdm-uu7zJt75z%2B_H-9AwjA%40mail.gmail.com.

Re: [go-nuts] CGO - Passing pointer to C

[Rick Hudson]

Breaking the Go CGO pointer rules comes up periodically and the rules 
have not changed. Applications have lived with the rules simply 
because breaking them results in revisiting the application code 
every time a new Go release comes out. Did the compiler improve and 
some object is now allocated on the stack instead of the heap? Did the 
runtime borrow some MESH [1] virtual memory page fragmentation 
techniques but improve them for Go by updating pointers to reducing 
TLB pressure? Is there value in moving objects from NVRAM to DRAM 
and updating pointers? And so forth and so on. Nobody knows if any of 
this will ever happen but the Go CGO pointer rules leave open the 
possibility.


[1] Bobby Powers, David Tench, Emery D. Berger, and Andrew McGregor. 2019. 
Mesh: Compacting Memory Management for C/C++ Applications. In 
Proceedings of the 40th ACM SIGPLAN Conference on Programming Language 
Design and Implementation (PLDI ’19), June 22ś26, 2019, Phoenix, AZ, USA. 
ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3314221.3314582


On Wednesday, December 4, 2019 at 11:14:17 AM UTC-5, Ian Lance Taylor wrote:
>
> On Wed, Dec 4, 2019 at 6:48 AM Robert Johnstone  > wrote: 
> > 
> > Thanks for the quick reply.  I had not considered the write barriers, 
> but if the Go objects are "live" longer than the held in C, it would work. 
>
> The write barriers do not look only at the pointer being stored, they 
> also look at the contents of memory being stored into.  That is why C 
> code must never store a Go pointer into Go memory. 
>
>
> > I definitely agree that there are risks associated with this approach. 
>  We are giving up some of the safety of Go.  Unfortunately, we are using 
> cgo for more than some computation, so we have live objects in C as well, 
> and so we cannot completely escape the manual memory management required. 
> > 
> > What is the state of plans for a moving garbage collector?  This would 
> definitely wreck havoc if any pointers to Go memory were held in C. 
>
> There are no current plans for a moving garbage collector. 
>
> I cannot promise that no other changes will break this approach. 
> Obviously we won't consider bug reports for code that breaks the cgo 
> rules. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/23885bd0-4802-40c7-b4c9-52541e92029f%40googlegroups.com.

Re: [go-nuts] CGO - Passing pointer to C

[Ian Lance Taylor]

On Wed, Dec 4, 2019 at 6:48 AM Robert Johnstone  wrote:
>
> Thanks for the quick reply.  I had not considered the write barriers, but if 
> the Go objects are "live" longer than the held in C, it would work.

The write barriers do not look only at the pointer being stored, they
also look at the contents of memory being stored into.  That is why C
code must never store a Go pointer into Go memory.


> I definitely agree that there are risks associated with this approach.  We 
> are giving up some of the safety of Go.  Unfortunately, we are using cgo for 
> more than some computation, so we have live objects in C as well, and so we 
> cannot completely escape the manual memory management required.
>
> What is the state of plans for a moving garbage collector?  This would 
> definitely wreck havoc if any pointers to Go memory were held in C.

There are no current plans for a moving garbage collector.

I cannot promise that no other changes will break this approach.
Obviously we won't consider bug reports for code that breaks the cgo
rules.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXq9k%2BzKd28Ff0fDQQYvbSC01fQnpxWe7Lk%3Djdaddtghg%40mail.gmail.com.

Re: [go-nuts] CGO - Passing pointer to C

[Robert Johnstone]

Hello,

Thanks for the quick reply.  I had not considered the write barriers, but 
if the Go objects are "live" longer than the held in C, it would work.

I definitely agree that there are risks associated with this approach.  We 
are giving up some of the safety of Go.  Unfortunately, we are using cgo 
for more than some computation, so we have live objects in C as well, and 
so we cannot completely escape the manual memory management required.

What is the state of plans for a moving garbage collector?  This would 
definitely wreck havoc if any pointers to Go memory were held in C.

Thank-you,

Robert


On Tuesday, 3 December 2019 22:39:35 UTC-5, Ian Lance Taylor wrote:
>
> On Tue, Dec 3, 2019 at 7:20 PM Robert Johnstone  > wrote: 
> > 
> > The section on passing pointer to C in the cgo documentation is quite 
> clear that letting C hold pointer to Go memory is not allowed.  I'd like to 
> better understand the limitation.  Frankly, because of the architecture of 
> a system I'm working on, the inability to transport pointers is 
> complicating matters. 
> > 
> > 1) The Go objects in question are certainly on the heap. 
> > 2) The references in C are not needed to maintain the lifetime of the Go 
> objects. 
> > 
> > My understanding is that the main reason for the restriction is that Go 
> might move to a moving garbage collector.  At the moment, unless there is 
> definite work on this, I'm inclined to break the rule and let the C code 
> hold onto pointers to Go memory.  However, before committing to this 
> approach, I'd like to make sure I haven't missed any other reasons for the 
> prohibition. 
>
> There is more background on this at 
> https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md 
> . 
>
> The Go compiler compiles code to use a write barrier when storing 
> pointers to memory.  The C compiler obviously does not.  When storing 
> a Go pointer in C code, no write barrier is executed.  This can cause 
> the garbage collector to be confused about whether the pointer is 
> reachable.  In particular, it is absolutely critical that C code never 
> store a Go pointer into Go memory. 
>
> Still, while you have to be careful and I recommend against it, you 
> may be able to get away with letting C code hold onto Go pointers 
> across calls, in C memory.  The Go values must be in the heap, not the 
> stack.  The Go values must be kept alive by existing Go pointers. 
>
> But it's risky and I personally would never do it.  Much better, if 
> possible, to use a handle.  Store the Go pointer in a map on the Go 
> side, and pass the map key, an integer, to the C side.  Then the C 
> side can pass the integer back, and Go code look it up in the map to 
> fetch the pointer. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c7f5f3b9-1811-4a45-9495-7337695990e6%40googlegroups.com.

Re: [go-nuts] CGO - Passing pointer to C

[Ian Lance Taylor]

On Tue, Dec 3, 2019 at 7:20 PM Robert Johnstone  wrote:
>
> The section on passing pointer to C in the cgo documentation is quite clear 
> that letting C hold pointer to Go memory is not allowed.  I'd like to better 
> understand the limitation.  Frankly, because of the architecture of a system 
> I'm working on, the inability to transport pointers is complicating matters.
>
> 1) The Go objects in question are certainly on the heap.
> 2) The references in C are not needed to maintain the lifetime of the Go 
> objects.
>
> My understanding is that the main reason for the restriction is that Go might 
> move to a moving garbage collector.  At the moment, unless there is definite 
> work on this, I'm inclined to break the rule and let the C code hold onto 
> pointers to Go memory.  However, before committing to this approach, I'd like 
> to make sure I haven't missed any other reasons for the prohibition.

There is more background on this at
https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md
.

The Go compiler compiles code to use a write barrier when storing
pointers to memory.  The C compiler obviously does not.  When storing
a Go pointer in C code, no write barrier is executed.  This can cause
the garbage collector to be confused about whether the pointer is
reachable.  In particular, it is absolutely critical that C code never
store a Go pointer into Go memory.

Still, while you have to be careful and I recommend against it, you
may be able to get away with letting C code hold onto Go pointers
across calls, in C memory.  The Go values must be in the heap, not the
stack.  The Go values must be kept alive by existing Go pointers.

But it's risky and I personally would never do it.  Much better, if
possible, to use a handle.  Store the Go pointer in a map on the Go
side, and pass the map key, an integer, to the C side.  Then the C
side can pass the integer back, and Go code look it up in the map to
fetch the pointer.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUCHrLtZrx1K418U_xVKCAxNM_A%2B%2B5-N4L7vUtN8FmO5w%40mail.gmail.com.