Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
Hi, Thanks for your recommendations, very interesting implementation :) I solved the problem with a callback from c to go to c. cheers, changkun On Saturday, September 29, 2018 at 6:27:08 PM UTC+2, ohir wrote: > > On Fri, 28 Sep 2018 22:27:04 -0700 (PDT) > changkun > wrote: > > > Not really, the original C code calls pango in every new created thread. > > [...] > > the Pango call is always called in a newly created thread. > > IIUC its your old C code spinning a new thread for each Pango > call (?). If so, you should not bridge such code directly with Go. You > need > to make a wrapper on the C side (in C), init it from the **single** > goroutine > you pin to a single (go side) thread then communicate only via said > wrapper calls. > > [Go] <-go-channels-> [comm goroutine] <-cgo-call(s)-> [wrapper] <-C-> > [library] > > You can try to leverage existing C implementation of Go channels for > the wrapper: > > https://github.com/tylertreat/chan > > [IDNK if these are still functioning with go1.11, ofc] > > Example: > https://github.com/matiasinsaurralde/cgo-channels > > > Best, > > changkun > > > > Hope this helps, > > -- > Wojciech S. Czarnecki > << ^oo^ >> OHIR-RIPE > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
Hi,
sorry for late response. Your solution is very inspiration! I solved the
problem
with letting pango call entering go domain and lock on os thread, then back
to c domain.
Thank you very much!
cheers, changkun
On Saturday, September 29, 2018 at 6:16:10 PM UTC+2, Tamás Gulácsi wrote:
>
>
> 2018. szeptember 29., szombat 17:10:25 UTC+2 időpontban changkun a
> következőt írta:
>>
>>
>>
>> On Saturday, September 29, 2018 at 4:03:42 PM UTC+2, Tamás Gulácsi wrote:
>>>
>>> Create One (1) goroutine with LockOSThread and let it get the todo and
>>> response channel from a channel and send the response back when ready, and
>>> make all other goroutines communicate with this one and only one
>>> pango-dedicated goroutine. Which is locked to a thread.
>>>
>>
>> Can't do that at the moment, reason in above.
>>
>
> I tought something like:
>
> type todo struct {
> FuncToCall string
> Args []interface{}
> Response ch chan<- response
> }
> type response struct {
> ResultCode int
> Err error
> }
>
> var todoCh = make(chan todo)
>
> go func(ch <- todo) {
> runtime.LockOSThread()
> defer runtime.UnlockOSThread()
> for t := range ch {
> switch t.FuncToCall {
> ...
> }
> t.Response <- response{Err:nil, ResultCode:0}
> }
> }()
>
> func handleConnection(timeout) error {
> ch := make(chan response, 1)
> ctx, cancel := context.WithTimeout(context.Background(), timeout)
> defer cancel()
> select {
> case <-ctx.Done():
> // timeout
> return ctx.Err()
> case todoCh <- todo{FuncToCall:..., Response:ch}:
> select {
> case <-ctx.Done():
> return ctx.Err()
> case resp := <-ch:
> // success!
> return nil
> }
> }
> }
>
>
> I don't feel that abstracting out the C (Pango) calls into an interface
> with maybe several implementations (C or Go) would be such a great
> undertaking,
> but would ease up writing correct code.
>
> And anyway you have to call that Pango thing from one thread.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Fri, 28 Sep 2018 22:27:04 -0700 (PDT) changkun wrote: > Not really, the original C code calls pango in every new created thread. > [...] > the Pango call is always called in a newly created thread. IIUC its your old C code spinning a new thread for each Pango call (?). If so, you should not bridge such code directly with Go. You need to make a wrapper on the C side (in C), init it from the **single** goroutine you pin to a single (go side) thread then communicate only via said wrapper calls. [Go] <-go-channels-> [comm goroutine] <-cgo-call(s)-> [wrapper] <-C-> [library] You can try to leverage existing C implementation of Go channels for the wrapper: https://github.com/tylertreat/chan [IDNK if these are still functioning with go1.11, ofc] Example: https://github.com/matiasinsaurralde/cgo-channels > Best, > changkun > Hope this helps, -- Wojciech S. Czarnecki << ^oo^ >> OHIR-RIPE -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
2018. szeptember 29., szombat 17:10:25 UTC+2 időpontban changkun a
következőt írta:
>
>
>
> On Saturday, September 29, 2018 at 4:03:42 PM UTC+2, Tamás Gulácsi wrote:
>>
>> Create One (1) goroutine with LockOSThread and let it get the todo and
>> response channel from a channel and send the response back when ready, and
>> make all other goroutines communicate with this one and only one
>> pango-dedicated goroutine. Which is locked to a thread.
>>
>
> Can't do that at the moment, reason in above.
>
I tought something like:
type todo struct {
FuncToCall string
Args []interface{}
Response ch chan<- response
}
type response struct {
ResultCode int
Err error
}
var todoCh = make(chan todo)
go func(ch <- todo) {
runtime.LockOSThread()
defer runtime.UnlockOSThread()
for t := range ch {
switch t.FuncToCall {
...
}
t.Response <- response{Err:nil, ResultCode:0}
}
}()
func handleConnection(timeout) error {
ch := make(chan response, 1)
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
select {
case <-ctx.Done():
// timeout
return ctx.Err()
case todoCh <- todo{FuncToCall:..., Response:ch}:
select {
case <-ctx.Done():
return ctx.Err()
case resp := <-ch:
// success!
return nil
}
}
}
I don't feel that abstracting out the C (Pango) calls into an interface
with maybe several implementations (C or Go) would be such a great
undertaking,
but would ease up writing correct code.
And anyway you have to call that Pango thing from one thread.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Saturday, September 29, 2018 at 4:03:42 PM UTC+2, Tamás Gulácsi wrote: > > Create One (1) goroutine with LockOSThread and let it get the todo and > response channel from a channel and send the response back when ready, and > make all other goroutines communicate with this one and only one > pango-dedicated goroutine. Which is locked to a thread. > Can't do that at the moment, reason in above. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Saturday, September 29, 2018 at 3:46:41 PM UTC+2, Robert Engels wrote: > > Your description makes no sense. Why are you creating another C thread, > you should just lock the OS thread the Go routine is on. Otherwise if you > create another C thread, in the cgo call you need to block that thread or > somehow communicate back into Go some other way - otherwise what is the > point? > As I mentioned on the top floor, I am migrating a very old pure C project to Go, and eventually become a pure Go project. the original project was linked by different components, and I am replacing them one by one... I know it doesn't make sense (apparently, before creating a non-Go thread, there are some internal preprocessing they are irrelevant to Pango) but to archive what you describe (in one goroutine) will consume loads of time for reading, understanding, and implementation, time is limited. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
Create One (1) goroutine with LockOSThread and let it get the todo and response channel from a channel and send the response back when ready, and make all other goroutines communicate with this one and only one pango-dedicated goroutine. Which is locked to a thread. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
Your description makes no sense. Why are you creating another C thread, you
should just lock the OS thread the Go routine is on. Otherwise if you create
another C thread, in the cgo call you need to block that thread or somehow
communicate back into Go some other way - otherwise what is the point?
> On Sep 29, 2018, at 7:31 AM, changkun wrote:
>
>
>> On Saturday, September 29, 2018 at 2:08:03 PM UTC+2, Tamás Gulácsi wrote:
>> Yes, but is that a one and only goroutine?
>
> No. The cgo call is opened for every new incoming user request.
>
> Let's summarize:
>
> - Every network request creates a goroutine without response
> processing result to a user of that goroutine.
> - The goroutine instantly proceeds a cgo call and the cgo call
> creates a non-Go thread, then a Pango call pango_font_map_get_default()
> is involved in the non-Go thread.
> - According to pango_font_map_get_default(), it holds a static thread-safe
> variable.
> - The original pure C code is able to proceed execution without involving Go.
> But stuck at the Pango call when involving cgo
> - runtime.LockOSThread doesn't work:
> go func() {
> runtime.LockOSThread()
> handleConnection(timeout)
> runtime.UnlockOSThread()
> }()
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Saturday, September 29, 2018 at 2:08:03 PM UTC+2, Tamás Gulácsi wrote:
>
> Yes, but is that a one and only goroutine?
No. The cgo call is opened for every new incoming user request.
Let's summarize:
- Every network request creates a goroutine without response
processing result to a user of that goroutine.
- The goroutine instantly proceeds a cgo call and the cgo call
creates a non-Go thread, then a Pango call pango_font_map_get_default()
is involved in the non-Go thread.
- According to pango_font_map_get_default(), it holds a static thread-safe
variable.
- The original pure C code is able to proceed execution without involving
Go.
But stuck at the Pango call when involving cgo
- runtime.LockOSThread doesn't work:
go func() {
runtime.LockOSThread()
handleConnection(timeout)
runtime.UnlockOSThread()
}()
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
Yes, but is that a one and only goroutine? -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
As discussed before, the every pango call is called in a non-Go newly created thread by C. runtime.LockOSThread doesn't work. On Saturday, September 29, 2018 at 1:24:21 PM UTC+2, Tamás Gulácsi wrote: > > You should LockOSThread and process every pango-facing call in that > goroutine. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
You should LockOSThread and process every pango-facing call in that goroutine. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Friday, September 28, 2018 at 6:44:10 PM UTC+2, Ian Lance Taylor wrote: > > On Fri, Sep 28, 2018 at 7:45 AM, changkun > wrote: > > > > On Friday, September 28, 2018 at 4:12:31 PM UTC+2, Ian Lance Taylor > wrote: > >> > >> On Fri, Sep 28, 2018 at 7:08 AM, changkun wrote: > >> > > >> > 1. Is my suspicion reasonable and correct? > >> > >> I wouldn't be my first guess. You say that pango memory is > >> per-thread. That suggests that you need to always call pango on a > >> consistent thread. > > > > > > Not really, the original C code calls pango in every new created thread. > > > > After involving cgo, it doesn’t change anything of this calling > behavior. > > Right, but what changes using cgo is that consecutive calls using the > same pango objects will be on different threads. That likely does not > happen in the original C code. > Yep. I understand that goroutines run in different threads. They seem not able to influence the entire C code. > > > >> That will not happen by default. You likely need > >> to arrange to make all your pango calls from a single goroutine, and > >> have that goroutine first call runtime.LockOSThread. > > > > > > Tried, it doesn’t work. A goroutine is created for the cgo call purpose, > > which instantly entering C side, when C code creates a thread, > > then call pango inside that thread. > > OK, sounds like I was wrong. But I don't understand what it means for > pango to have per-thread data structures if you always make pango > calls in a newly created thread. > That is actually the most wired part. Indeed, the Pango call is always called in a newly created thread. So my concern is somehow someway, cgo or Go influences the memory layout, then probably caused the "double variable registering" of pango underlying call to glib? > > > >> > 2. Why an error when involving cgo? > >> > >> My first guess would be that it is because you are calling the cgo > >> code from different threads. > > > > > > Nope. It is all about from Go to C then C doing something > > without sending data back to Go. I am not calling Go from C. > > No, I didn't think you were. > > > > > I was trying to express that, are memories in every non-Go thread > > are isolated one another? > > > > The warning from pango: > > > > (process:1): GLib-GObject-WARNING **: cannot register existing type > > 'PangoFontMap' > > > > Looks like that the pango call is double registering the font map > object. > > If the memory are shared across all non-Go thread, it cloud be an issue, > > isn’t it? > > All threads in both C and Go share the same memory space. > > But, obviously, they do not share the same per-thread data structures. > Perhaps, to understand where the problem comes from, the only way is to read what actually the call is doing, g_private_get() and pango_cairo_font_map_new() in GLib and Pango correspondingly. https://github.com/GNOME/pango/blob/2bb6262be19dc17f7b56d3ce36c0defdbdb55c97/pango/pangocairo-fontmap.c#L184 However, I rather say, GTKs are extremely stable for years, consider the problem arise when involving Go and cgo, and I do not change any of the C part, it is most likely the Go and cgo cause the problem :( Best, changkun -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Fri, Sep 28, 2018 at 7:45 AM, changkun wrote: > > On Friday, September 28, 2018 at 4:12:31 PM UTC+2, Ian Lance Taylor wrote: >> >> On Fri, Sep 28, 2018 at 7:08 AM, changkun wrote: >> > >> > 1. Is my suspicion reasonable and correct? >> >> I wouldn't be my first guess. You say that pango memory is >> per-thread. That suggests that you need to always call pango on a >> consistent thread. > > > Not really, the original C code calls pango in every new created thread. > > After involving cgo, it doesn’t change anything of this calling behavior. Right, but what changes using cgo is that consecutive calls using the same pango objects will be on different threads. That likely does not happen in the original C code. >> That will not happen by default. You likely need >> to arrange to make all your pango calls from a single goroutine, and >> have that goroutine first call runtime.LockOSThread. > > > Tried, it doesn’t work. A goroutine is created for the cgo call purpose, > which instantly entering C side, when C code creates a thread, > then call pango inside that thread. OK, sounds like I was wrong. But I don't understand what it means for pango to have per-thread data structures if you always make pango calls in a newly created thread. >> > 2. Why an error when involving cgo? >> >> My first guess would be that it is because you are calling the cgo >> code from different threads. > > > Nope. It is all about from Go to C then C doing something > without sending data back to Go. I am not calling Go from C. No, I didn't think you were. >> > 3. Are memories in a non-Go thread completely isolated to Go >> > memory per-non-Go thread? >> >> Sorry, I don't understand this question. >> > > I was trying to express that, are memories in every non-Go thread > are isolated one another? > > The warning from pango: > > (process:1): GLib-GObject-WARNING **: cannot register existing type > 'PangoFontMap' > > Looks like that the pango call is double registering the font map object. > If the memory are shared across all non-Go thread, it cloud be an issue, > isn’t it? All threads in both C and Go share the same memory space. But, obviously, they do not share the same per-thread data structures. Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Friday, September 28, 2018 at 4:12:31 PM UTC+2, Ian Lance Taylor wrote: > > On Fri, Sep 28, 2018 at 7:08 AM, changkun > wrote: > > > > 1. Is my suspicion reasonable and correct? > > I wouldn't be my first guess. You say that pango memory is > per-thread. That suggests that you need to always call pango on a > consistent thread. Not really, the original C code calls pango in every new created thread. After involving cgo, it doesn’t change anything of this calling behavior. The pango call is per-thread, according to its document, it is literally a per-thread singleton and only allocate in the first call in every different thread if I read the document correctly: https://developer.gnome.org/pango/stable/pango-Cairo-Rendering.html#pango-cairo-font-map-get-default > That will not happen by default. You likely need > to arrange to make all your pango calls from a single goroutine, and > have that goroutine first call runtime.LockOSThread. > Tried, it doesn’t work. A goroutine is created for the cgo call purpose, which instantly entering C side, when C code creates a thread, then call pango inside that thread. > > > 2. Why an error when involving cgo? > > My first guess would be that it is because you are calling the cgo > code from different threads. > Nope. It is all about from Go to C then C doing something without sending data back to Go. I am not calling Go from C. > > 3. Are memories in a non-Go thread completely isolated to Go > > memory per-non-Go thread? > > Sorry, I don't understand this question. > > I was trying to express that, are memories in every non-Go thread are isolated one another? The warning from pango: (process:1): GLib-GObject-WARNING **: cannot register existing type 'PangoFontMap' Looks like that the pango call is double registering the font map object. If the memory are shared across all non-Go thread, it cloud be an issue, isn’t it? Best, changkun -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Are C allocated memory completely isolated to Go memory in any cases?
On Fri, Sep 28, 2018 at 7:08 AM, changkun wrote: > > 1. Is my suspicion reasonable and correct? I wouldn't be my first guess. You say that pango memory is per-thread. That suggests that you need to always call pango on a consistent thread. That will not happen by default. You likely need to arrange to make all your pango calls from a single goroutine, and have that goroutine first call runtime.LockOSThread. > 2. Why an error when involving cgo? My first guess would be that it is because you are calling the cgo code from different threads. > 3. Are memories in a non-Go thread completely isolated to Go > memory per-non-Go thread? Sorry, I don't understand this question. > 4. Are C memories allocated in a cgo call always stands in > the same sharing heap even threading? If I understand you correctly, then, yes. Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
