[go-nuts] Congrats to the Go team

['Robert Engels' via golang-nuts]

I have a fairly stable project github.com/robaho/fixed 
 which is almost 100% cpu bound. It doesn’t 
change so it makes a great way to compare the performance of different Go 
versions using the same hardware. I took the time to re-run the tests today.

Using 1.21.17:

BenchmarkAddFixed-8 20   0.59 ns/op
0 B/op  0 allocs/op 

  
BenchmarkAddDecimal-8500   243 ns/op 
176 B/op  8 allocs/op   


BenchmarkAddBigInt-81   14.3 ns/op 
0 B/op  0 allocs/op 

  
BenchmarkAddBigFloat-8  200078.8 ns/op
48 B/op  1 allocs/op

   
BenchmarkMulFixed-8 34.88 ns/op
0 B/op  0 allocs/op 

  
BenchmarkMulDecimal-8   200072.0 ns/op
80 B/op  2 allocs/op

   
BenchmarkMulBigInt-81   17.1 ns/op 
0 B/op  0 allocs/op 

  
BenchmarkMulBigFloat-8  300035.5 ns/op 
0 B/op  0 allocs/op 

  
BenchmarkDivFixed-8 34.71 ns/op
0 B/op  0 allocs/op 

  
BenchmarkDivDecimal-8200   779 ns/op 
568 B/op 21 allocs/op   


BenchmarkDivBigInt-8300046.1 ns/op 
8 B/op  1 allocs/op 

  
BenchmarkDivBigFloat-8  2000   108 ns/op  
24 B/op  2 allocs/op

   
BenchmarkCmpFixed-8 20   0.38 ns/op
0 B/op  0 allocs/op 

  
BenchmarkCmpDecimal-8   28.05 ns/op
0 B/op  0 allocs/op 

  
BenchmarkCmpBigInt-835.87 ns/op
0 B/op  0 allocs/op 

  

Re: [go-nuts] Critical Section - static analysis

[robert engels]

The github link you provided results in a 404 error.

> On Apr 22, 2024, at 1:58 PM, Stephen Illingworth 
>  wrote:
> 
> Hello,
> 
> I've created a proof-of-concept for a method of defining critical sections.
> Crucially, the scheme allows for static analysis, thereby helping prevent
> violations of the critical sections.
> 
> It's a simple concept but I think it's something that can be built upon.
> 
> https://github.com/JetSetIlly/critsec
> 
> I'm posting it here because I think it's an interesting problem and I would 
> like
> to hear expert opinion. The problem is a real one but does this solution have 
> any
> merit?
> 
> -
> 
> The basic mechanism in this project is something I've used in a large rproject
> where performance is important. I wanted a way of locking access to data but
> without the risk of forgetting to unlock it. Very simple: 
> 
>   func (crit *Section) Lease(f func() error) error {
>   crit.lock.Lock()
>   defer crit.lock.Unlock()
>   return f()
>   }
>   
> I've been very happy with that but I thought it would be interesting to come 
> up
> with a method for statically analysing usage of the Lease() function. In other
> words, to detect those times when I access critical data outside of a Lease.
> 
> What's needed for this to work is a way of tagging data to say that it should
> only be accessed 'inside' a Lease().
> 
> My idea is this: define critical data by embedding a crit.Section struct in a
> new struct type:
> 
>   type Section struct {
>   lock sync.Mutex
>   }
> 
> 
>   type exampleCritSectioning struct {
>   crit.Section
>   a int
>   b bool
>   }
>   
> We can then lease the critical section:
> 
>   var C exampleCritSectioning
>   _ = C.Lease(func() error {
>   a = 100
>   return nil
>   })
> 
> 
> 
> So, critical data is identified with the crit.Section type and critical 
> sections
> by the entry and exit of the Lease function. Static analysis is now relatively
> simple:
> 
> 1. Find all instances of types that embed the crit.Section type
> 2. Find all accesses to data of those types
> 3. Check whether the callgraph to those access include a call to the Lease() 
> function
>
>
> The proof-of-concept project includes an analysis package as well as the crit
> package. A detailed example of an analysis report is given in the README
> 
> 
> Notes
> 
> I haven't used the analysistest package for testing because I had problems 
> with
> getting it to work with a go.work file. The examples/ package is used in lieue
> of that. 
> 
> A standalone driver for the analysis is in 'analysis/cmd/critcheck'
> 
> The callgraph is created with the 'x/tools/go/callgraph/vta' package. I 
> didn't put
> much though into this so there might be a better graph method. But VTA 
> provided
> the best results during experimentation
> 
> 
> Regards
> Stephen
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e2d9f1d9-958a-45df-b402-fe98a213fa67n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CB784B45-7498-4C12-98CC-07CEBCFE9DD0%40ix.netcom.com.

Re: [go-nuts] xml to json, parsing xml

[robert engels]

I don’t think that is true. There are multiple ways to model XML into json.

See this http://badgerfish.ning.com  for one of 
them.

> On Apr 23, 2024, at 11:43 AM, burak serdar  wrote:
> 
> In general, you cannot convert xml to json. They have incompatible
> models. XML elements are ordered similar to a JSON array, but in many
> cases you want to map XML elements to JSON objects, which are
> unordered name-value collections. Also, there is no JSON equivalent of
> an XML attribute.
> 
> If you want to work with XML, either use xml marshaling, or find a
> third-party DOM library.
> 
> On Tue, Apr 23, 2024 at 10:29 AM Don Caldwell  wrote:
>> 
>> Disclaimer - I am very new to golang.
>> I puzzled about this for a few days. After some struggle, I got a little 
>> program working that parses
>> arbitrary xml into a structure that json can understand. You can find it 
>> here:
>> https://github.com/dfwcnj/gxmldecode
>> 
>> On Thursday, October 7, 2021 at 10:06:30 AM UTC-4 RS wrote:
>>> 
>>> What is the best approach to convert xml to json?
>>> However I need above all to parse a very complex xml to see if related 
>>> properties are existing.
>>> I thought maybe converting it to Json maybe make it easier.
>>> 
>>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/90c0dd22-2d81-4393-b534-651a2376f386n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAMV2RqowOgbOnmGxsWegOuJ-_crQcNhsjj1Gxk3pAXhBmtNK5Q%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1415B2A9-7CB5-4D33-B47F-FB7992B7165D%40ix.netcom.com.

Re: [go-nuts] Re: Concurrent queries in a single transaction

[Robert Engels]

As you state, the driver (and db) need to support this. The synchronization for something like this is internal. But, if the operations are modifying the same tables you may end up with serialized operations anyway in order to ensure consistency while avoiding deadlocks. On Apr 20, 2024, at 12:11 AM, nilsocket  wrote:Does any solution exist for this problem?I'm facing similar problem, want to run bunch of queries inside a transaction,but want to run them concurrently, as running sequential is taking more than 1 second.On Friday, June 29, 2018 at 6:32:40 PM UTC+5:30 Space A. wrote:Hi,DB in common case will do lock/unlock when necessary for you implicitly. It is safe to call queries simultaneously. Moreover it is general approach in Enterprise solutions to have let's say hundreds of open connections sockets towards DB and execute statements from hundreds of threads. However to answer your question accurate you need to look at underlying implementation. As I understand, most of Go DB implementations don't even have connection pool capabilities. So they are even "safer" from that point because single network socket will become additional (and unnecessary) "serialization" mechanism. You could probably try to implement connection pool by yourself.Regards,среда, 27 июня 2018 г., 15:19:27 UTC+3 пользователь Ain написал:HiIt is my understanding that DB Tx is generally not safe to use concurrently, ie consider following code:    tx, err := db.Begin()    defer tx.Rollback()    ...    statements[0], err := tx.Prepare("...")    defer s1.Close()    statements[1], err := tx.Prepare("...")    defer s2.Close()    ...    wg.Add(len(statements))    for x := range statements {        go func(x int) {            defer wg.Done()            statements[x].QueryRow(...).Scan(...)        }(x)    }    wg.Wait()Here we prepare bunch of statements for a Tx which are "independent of each other", ie query different tables and scan to different variables etc. However they can't be executed concurrently, in a goroutine, as they share the same connection and thus interfere with each other?It might be safe when db driver takes care to serialise the statements but database/sql doesn't guarantee safe usage for given secenario.Is that correct?So how would you go about executing multiple queries in the same transaction context concurrenty?TIAain



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4c798067-55cd-4d20-9eba-875c4743ce73n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/B34F5737-742D-45F2-8917-8C60EC79BF6D%40ix.netcom.com.

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

Happy that it sparked an idea. I also don’t think Eli’s concern is valid. if there are other requests in flight (on different connections I assume) - let those continue - just put any new requests on a new transport for that host (after a 500 error is encountered) - then tear down the bad when it has no more requests being processed. On Apr 3, 2024, at 1:50 PM, Jim Minter  wrote:Yes, I agree, I think this approach makes sense (and should have been obvious to me as something to try...).  It could be implementable as a wrapper transport too.  I'll try it out and reply back here if it doesn't work.Thank-you!JimOn Wed, 3 Apr 2024 at 12:46, Robert Engels <reng...@ix.netcom.com> wrote:Just create a recyclable transport for the bad server and put all of the rest on a single shared transport. If one connection is returning 500 for all requests I can’t see how a different connection would solve that - unless the backend is completely broken. On Apr 3, 2024, at 7:48 AM, Eli Lindsey <e...@siliconsprawl.com> wrote:It would work, but has potentially high cost since it also causes any healthy conns in the pool to be torn down. How useful it is in practice depends on request rate, number of backends behind the lb, and ratio of healthy to unhealthy (500’ing) connections. It’s hard to tell from the description if it would work here - retrying and reusing the same busted connection could mean that the request rate is very low and there’s only one idle conn (in which case cycling the transport is a good solution), or it could mean that the unhealthy conn is quicker to respond than the pooled healthy conns and gobbles up a disproportionate share of requests.Tangential question, when the backend servers land in this state does the lb not detect and remove them?
-eli


On Apr 3, 2024, at 6:41 AM, Robert Engels <reng...@ix.netcom.com> wrote:That probably wasn’t clear. Why not create a Transport per host. Then when the 500 is encountered stop using that transport completely and create a new instance. Probably want to cancel any requests currently in flight. The connection pool is per transport. On Apr 2, 2024, at 11:05 PM, Eli Lindsey <e...@siliconsprawl.com> wrote:There isn’t a great way to handle this currently - we maintain out of tree patches to do something similar, though ours are h2 specific. The crux of the problem is that net currently lacks a usable connection pool API (there is some slightly newer discussion here, but it’s similar to the issue you linked https://github.com/golang/go/discussions/60746).If you want to stay in tree, one option may be using httptrace GotConnInfo and calling Close on the underlying connection (in direct violation of GotConnInfo’s doc). I would expect this to error out anything inflight, but otherwise be benign (though I have not checked :) ).
-eli


On Apr 2, 2024, at 3:29 PM, Jim Minter <j...@minter.uk> wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not sure how else this could be done.  Does anyone have any thoughts?Many thanks 

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

Just create a recyclable transport for the bad server and put all of the rest on a single shared transport. If one connection is returning 500 for all requests I can’t see how a different connection would solve that - unless the backend is completely broken. On Apr 3, 2024, at 7:48 AM, Eli Lindsey  wrote:It would work, but has potentially high cost since it also causes any healthy conns in the pool to be torn down. How useful it is in practice depends on request rate, number of backends behind the lb, and ratio of healthy to unhealthy (500’ing) connections. It’s hard to tell from the description if it would work here - retrying and reusing the same busted connection could mean that the request rate is very low and there’s only one idle conn (in which case cycling the transport is a good solution), or it could mean that the unhealthy conn is quicker to respond than the pooled healthy conns and gobbles up a disproportionate share of requests.Tangential question, when the backend servers land in this state does the lb not detect and remove them?
-eli


On Apr 3, 2024, at 6:41 AM, Robert Engels  wrote:That probably wasn’t clear. Why not create a Transport per host. Then when the 500 is encountered stop using that transport completely and create a new instance. Probably want to cancel any requests currently in flight. The connection pool is per transport. On Apr 2, 2024, at 11:05 PM, Eli Lindsey  wrote:There isn’t a great way to handle this currently - we maintain out of tree patches to do something similar, though ours are h2 specific. The crux of the problem is that net currently lacks a usable connection pool API (there is some slightly newer discussion here, but it’s similar to the issue you linked https://github.com/golang/go/discussions/60746).If you want to stay in tree, one option may be using httptrace GotConnInfo and calling Close on the underlying connection (in direct violation of GotConnInfo’s doc). I would expect this to error out anything inflight, but otherwise be benign (though I have not checked :) ).
-eli


On Apr 2, 2024, at 3:29 PM, Jim Minter  wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not sure how else this could be done.  Does anyone have any thoughts?Many thanks for the help,Jim

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/34d597cf-a84c-48eb-b555-537a8768f468n%40googlegroups.com.


-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/080B6923-51DA-4DDB-9400-B1054C1DFCE4%40siliconsprawl.com.




-- 
You received this

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

That probably wasn’t clear. Why not create a Transport per host. Then when the 500 is encountered stop using that transport completely and create a new instance. Probably want to cancel any requests currently in flight. The connection pool is per transport. On Apr 2, 2024, at 11:05 PM, Eli Lindsey  wrote:There isn’t a great way to handle this currently - we maintain out of tree patches to do something similar, though ours are h2 specific. The crux of the problem is that net currently lacks a usable connection pool API (there is some slightly newer discussion here, but it’s similar to the issue you linked https://github.com/golang/go/discussions/60746).If you want to stay in tree, one option may be using httptrace GotConnInfo and calling Close on the underlying connection (in direct violation of GotConnInfo’s doc). I would expect this to error out anything inflight, but otherwise be benign (though I have not checked :) ).
-eli


On Apr 2, 2024, at 3:29 PM, Jim Minter  wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not sure how else this could be done.  Does anyone have any thoughts?Many thanks for the help,Jim

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/34d597cf-a84c-48eb-b555-537a8768f468n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/080B6923-51DA-4DDB-9400-B1054C1DFCE4%40siliconsprawl.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6AA93658-082F-4580-A3BD-6603D1D83394%40ix.netcom.com.

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

My guess is that if you are getting a 500 you have exhausted the server or capacity. So close the client completely and perform an exponential back off. You can wrap all of this at a higher level to keep the synchronous behavior. On Apr 2, 2024, at 7:37 PM, Jim Minter  wrote:That's possible, but the rate of occurrence of the issue is low (but painful when it happens), and the costs of starting a new TLS connection for every HTTP request are significant.  I'm looking for a better way.JimOn Tuesday 2 April 2024 at 15:01:30 UTC-6 Sean Liao wrote:since you already know the server is problematic, you could just set Close on the original request.On Tue, Apr 2, 2024, 15:29 Jim Minter  wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not sure how else this could be done.  Does anyone have any thoughts?Many thanks for the help,Jim



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/34d597cf-a84c-48eb-b555-537a8768f468n%40googlegroups.com.
- sean




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a3d208a6-90c0-42af-9a13-a8f3f0c7b21dn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1180CF7E-4287-4BD2-85A3-E91ABD1CF650%40ix.netcom.com.

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

Apologies.On Mar 9, 2024, at 1:16 PM, Ian Lance Taylor  wrote:This thread has gotten heated.  I think it's time to move on to different topics and give this discussion a rest.  Thanks.Ian



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXbXBEMCpgPH_piQH0-jZoC_Lok94Oo9%2Bbe2fDBnaWgNA%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/26C7753E-CEF8-42F9-9ED6-68513CC0BF5A%40ix.netcom.com.

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

My guess is that most applications are decoding 25gb json files with any regularity. Even transferring 25 GB over the fastest of networks takes 20 secs? So that reduces the cost to less than 10%???The test isn’t doing anything with the decoded json - maybe Go is 1000% faster in that. This is either a troll post or by someone that needs more education in performance monitoring. Sorry. On Mar 8, 2024, at 10:25 PM, Mike Schinkel  wrote:Hi Robert,Thanks for the reply.I decided to run a quick benchmark so I compared Go 1.22's json.UnMarshal(any) using an any type with PHP 8.3.2's json_decode()and found that PHP's option is 85% faster for a ~25Mb JSON file.  Optimizing it with a struct I was able to get Go down to being only about 25% slower than PHP's  json_decode().Note that they are very rough benchmarks, probably only valid in gauging magnitude, not specific performance.I wrote up the results and provided the code in this gist for anyone who might be interested.  For me, the takeaway is that if you really need fast JSON unmarshalling you probably need to look to one of the 3rd party JSON parser packages.Whether or not the extra time it takes Go to parse JSON vs. PHP is actually relevant for her use-case remains to be seen given the potential order-of-magnitude difference in web request latency.  Still, given that she seems to be controlling for that web request latency by comparing her PHP API to her Go API would indicate — sans issues like buffering or slow Go middleware — that the performance of JSON parsing made indeed by one of the reasons she is seeing a difference.Pragya, it would be really nice if you could follow up to close the loop and let us know what the actual bottleneck was and how you ended up solving it.-MikeOn Friday, March 8, 2024 at 9:58:26 PM UTC-5 Robert Engels wrote:Related, 30k of anything is largely meaningless for modern hardware in terms of cpu processing - where billions of operations a second is the norm. On Mar 8, 2024, at 8:55 PM, Robert Engels <ren...@ix.netcom.com> wrote:The point I was trying to make - maybe unclear - is that it is doubtful any raw performance differences between PHP and Go in terms of json encoding/decoding would be noticeable- unless it was a hft/hpc system which would be doubtful given that the original system was PHP. So if there is detectable difference in performance it is almost certainly related to IO management concerns (buffering, window sizes, etc)Without detailed measurements along with how they were measured it’s hard to know with any certainty- so this was a Large guess to help you know where to look. On Mar 8, 2024, at 8:33 PM, Mike Schinkel <mi...@newclarity.net> wrote:Hi Robert,I am now confused.  While your original reply to this thread made perfect sense to me, your follow up for the purposes of clarification had the opposite effect, at least for me. I am not clear the point you are trying to make.You mention the difference between compiled and interpreted and point out that typically there is not a large performance difference — where I assume you were referring to use of standard library functions in the interpreted language such as json_decode() in PHP. That is of course often true and I concur. (Of course, try writing a full JSON parser in pure PHP and we'll find a big difference in performance with a compiled language like Go.) But I am not seeing why that point is relevant in the thread because she was not asking "Why is PHP code slower than Go code?" in which case I would understand why you chose to make that distinction.You also mention that IO performance (and design(?)) over the web is typically much more relevant performance-wise, on which I also concur, but given that both her PHP and her Go API endpoints presumably had much the same web-based latency and response time concerns, I'm struggling to understand why it was relevant to state it in this context.  Of course your point simply could have been not using a buffered output stream,  but you already mentioned, and your follow up did not make that connection clear.So, can you please help me better understand the point you were trying to make?  As a follow up to my reply were you trying to imply that illustrating ways to optimize JSON parsing was something you felt I should not have posted in response to her question? That feels like in might have been your intent, but I could certainly have misinterpreted and if so would prefer to know rather than wrongly assume.-MikeOn Friday, March 8, 2024 at 8:28:38 PM UTC-5 Robert Engels wrote:Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC systems deviate from this 

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

Related, 30k of anything is largely meaningless for modern hardware in terms of cpu processing - where billions of operations a second is the norm. On Mar 8, 2024, at 8:55 PM, Robert Engels  wrote:The point I was trying to make - maybe unclear - is that it is doubtful any raw performance differences between PHP and Go in terms of json encoding/decoding would be noticeable- unless it was a hft/hpc system which would be doubtful given that the original system was PHP. So if there is detectable difference in performance it is almost certainly related to IO management concerns (buffering, window sizes, etc)Without detailed measurements along with how they were measured it’s hard to know with any certainty- so this was a Large guess to help you know where to look. On Mar 8, 2024, at 8:33 PM, Mike Schinkel  wrote:Hi Robert,I am now confused.  While your original reply to this thread made perfect sense to me, your follow up for the purposes of clarification had the opposite effect, at least for me. I am not clear the point you are trying to make.You mention the difference between compiled and interpreted and point out that typically there is not a large performance difference — where I assume you were referring to use of standard library functions in the interpreted language such as json_decode() in PHP. That is of course often true and I concur. (Of course, try writing a full JSON parser in pure PHP and we'll find a big difference in performance with a compiled language like Go.) But I am not seeing why that point is relevant in the thread because she was not asking "Why is PHP code slower than Go code?" in which case I would understand why you chose to make that distinction.You also mention that IO performance (and design(?)) over the web is typically much more relevant performance-wise, on which I also concur, but given that both her PHP and her Go API endpoints presumably had much the same web-based latency and response time concerns, I'm struggling to understand why it was relevant to state it in this context.  Of course your point simply could have been not using a buffered output stream,  but you already mentioned, and your follow up did not make that connection clear.So, can you please help me better understand the point you were trying to make?  As a follow up to my reply were you trying to imply that illustrating ways to optimize JSON parsing was something you felt I should not have posted in response to her question? That feels like in might have been your intent, but I could certainly have misinterpreted and if so would prefer to know rather than wrongly assume.-MikeOn Friday, March 8, 2024 at 8:28:38 PM UTC-5 Robert Engels wrote:Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC systems deviate from this but by the description and reference to the previous implementation I doubt that is the case here. On Mar 8, 2024, at 6:53 PM, Mike Schinkel <mi...@newclarity.net> wrote:Hi Pragya,While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things you can look at.1. You mention you have to use a map because of response keys not being fixed. Be aware that you do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need, for example.  2. If you optionally need to parse portions of the JSON, but not always, you can use json.RawMessage to capture the properties you don't always need to parse, and then parse them only when you need to.3. You can also declare an .UnmarshalJSON() method on a struct you are passing to json.Unmarshal(), or on any struct that is a property of your top level object, and then be fully in control of parsing the data, meaning you could combine with (a) json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to.4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library.  Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole.  However, if you really need better performance it might be worth the added dependency.While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance:https://github.com/valyala/fastjsonhttps://github.com/bytedance/sonicht

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

The point I was trying to make - maybe unclear - is that it is doubtful any raw performance differences between PHP and Go in terms of json encoding/decoding would be noticeable- unless it was a hft/hpc system which would be doubtful given that the original system was PHP. So if there is detectable difference in performance it is almost certainly related to IO management concerns (buffering, window sizes, etc)Without detailed measurements along with how they were measured it’s hard to know with any certainty- so this was a Large guess to help you know where to look. On Mar 8, 2024, at 8:33 PM, Mike Schinkel  wrote:Hi Robert,I am now confused.  While your original reply to this thread made perfect sense to me, your follow up for the purposes of clarification had the opposite effect, at least for me. I am not clear the point you are trying to make.You mention the difference between compiled and interpreted and point out that typically there is not a large performance difference — where I assume you were referring to use of standard library functions in the interpreted language such as json_decode() in PHP. That is of course often true and I concur. (Of course, try writing a full JSON parser in pure PHP and we'll find a big difference in performance with a compiled language like Go.) But I am not seeing why that point is relevant in the thread because she was not asking "Why is PHP code slower than Go code?" in which case I would understand why you chose to make that distinction.You also mention that IO performance (and design(?)) over the web is typically much more relevant performance-wise, on which I also concur, but given that both her PHP and her Go API endpoints presumably had much the same web-based latency and response time concerns, I'm struggling to understand why it was relevant to state it in this context.  Of course your point simply could have been not using a buffered output stream,  but you already mentioned, and your follow up did not make that connection clear.So, can you please help me better understand the point you were trying to make?  As a follow up to my reply were you trying to imply that illustrating ways to optimize JSON parsing was something you felt I should not have posted in response to her question? That feels like in might have been your intent, but I could certainly have misinterpreted and if so would prefer to know rather than wrongly assume.-MikeOn Friday, March 8, 2024 at 8:28:38 PM UTC-5 Robert Engels wrote:Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC systems deviate from this but by the description and reference to the previous implementation I doubt that is the case here. On Mar 8, 2024, at 6:53 PM, Mike Schinkel <mi...@newclarity.net> wrote:Hi Pragya,While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things you can look at.1. You mention you have to use a map because of response keys not being fixed. Be aware that you do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need, for example.  2. If you optionally need to parse portions of the JSON, but not always, you can use json.RawMessage to capture the properties you don't always need to parse, and then parse them only when you need to.3. You can also declare an .UnmarshalJSON() method on a struct you are passing to json.Unmarshal(), or on any struct that is a property of your top level object, and then be fully in control of parsing the data, meaning you could combine with (a) json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to.4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library.  Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole.  However, if you really need better performance it might be worth the added dependency.While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance:https://github.com/valyala/fastjsonhttps://github.com/bytedance/sonichttps://github.com/buger/jsonparserhttps://github.com/mailru/easyjsonhttps://github.com/sugawarayuuta/sonnethttps://github.com/json-iterator/gohttps://github.com/segmentio/encodinghttps://github.com/go

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC systems deviate from this but by the description and reference to the previous implementation I doubt that is the case here. On Mar 8, 2024, at 6:53 PM, Mike Schinkel  wrote:Hi Pragya,While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things you can look at.1. You mention you have to use a map because of response keys not being fixed. Be aware that you do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need, for example.  2. If you optionally need to parse portions of the JSON, but not always, you can use json.RawMessage to capture the properties you don't always need to parse, and then parse them only when you need to.3. You can also declare an .UnmarshalJSON() method on a struct you are passing to json.Unmarshal(), or on any struct that is a property of your top level object, and then be fully in control of parsing the data, meaning you could combine with (a) json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to.4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library.  Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole.  However, if you really need better performance it might be worth the added dependency.While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance:https://github.com/valyala/fastjsonhttps://github.com/bytedance/sonichttps://github.com/buger/jsonparserhttps://github.com/mailru/easyjsonhttps://github.com/sugawarayuuta/sonnethttps://github.com/json-iterator/gohttps://github.com/segmentio/encodinghttps://github.com/goccy/go-jsonhttps://github.com/simdjson/simdjsonBenchmarks from authors of some of the packages (so take with a grain of salt):https://github.com/sugawarayuuta/benchmarkhttps://blog.min.io/simdjson-go-parsing-gigabyes-of-json-per-second-in-go/And benchmarks from someone who is not an author of one of those packages:https://kokizzu.blogspot.com/2022/12/map-to-struct-and-struct-to-map-golang.html Hope this helps.-MikeOn Friday, March 8, 2024 at 4:15:25 PM UTC-5 Robert Engels wrote:It is highly unlikely that the Go marshaling is the cause. I’m guessing you are probably not using a buffered output stream. PHP is written in C but depending on what you are doing it is either a script or calling out to another process (afaik) On Mar 8, 2024, at 2:35 PM, pragya singh <pragy...@gmail.com> wrote:Hi,"I am facing an issue in my Go code. I converted my read API from PHP to Go, but the response time of Go API is higher than PHP. We are returning the response in JSON format, and the response contains around 30k keys, which is taking too much time in JSON marshaling. We are adding all the response in map format as we can't use struct because response keys are not fixed and vary from user to user.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a57a2a71-b0f9-41fb-93e9-bd54b57829b1n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7345c383-1529-489c-b156-72601bd80530n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/C19386B5-82FB-489A-896B-06807B09C6CD%40ix.netcom.com.

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

It is highly unlikely that the Go marshaling is the cause. I’m guessing you are probably not using a buffered output stream. PHP is written in C but depending on what you are doing it is either a script or calling out to another process (afaik) On Mar 8, 2024, at 2:35 PM, pragya singh  wrote:Hi,"I am facing an issue in my Go code. I converted my read API from PHP to Go, but the response time of Go API is higher than PHP. We are returning the response in JSON format, and the response contains around 30k keys, which is taking too much time in JSON marshaling. We are adding all the response in map format as we can't use struct because response keys are not fixed and vary from user to user.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a57a2a71-b0f9-41fb-93e9-bd54b57829b1n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/32B1DA04-4F39-40B7-8A70-CB5A669248D2%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

Glad you figured it out. Not certain how requests could crash a process like… I think you’d be better off configuring a maximum heap size rather than having the OOM killer kick in On Mar 2, 2024, at 3:50 PM, Russtopia  wrote:SOLVED!Thank you all for the helpful suggestions. Although it has turned out to be something totally different, and a teachable lesson in web app design...This go tool of mine has a very simple web interface with controls for a set of jobs on the main page.The jobs on this list can be run, viewed, and most importantly, cancelled via an endpoint of the form "/cancel/?id=nnn" ...I have had the site up in a "demo" mode on the public internet at various times, including recently -- it turns out that very recently some crawl bots must have found it, and they are following those /cancel/ links on the dashboard almost as soon as they appear -- they must be scanning at <5sec intervals to find the new unique jobIDs encoded in each 'cancel' link. Oops :)The app's usually behind an auth page, but this time it wasn't. I'm not a web dev so rookie mistake I suppose!I guess I really should have a 'robots.txt' file served out by my go app to prevent this, and perhaps consider other client session ids to prevent outside crawlers from accidentally activating my app's link endpoints.Thank you again, all.On Sat, Mar 2, 2024 at 6:23 PM Robert Engels <reng...@ix.netcom.com> wrote:I would  be also try reverting the Go version and ensure that it continues to work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor <i...@golang.org> wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia <rma...@gmail.com> wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.





-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/852FA6F1-8B90-4361-B1BC-579FBEC83513%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

I’m guessing some other library or package you installed has corrupted your Linux installation. Sadly, my suggestion would be a fresh install of Linux. On Mar 2, 2024, at 1:58 PM, Russtopia  wrote:It no longer does.. so it suggests to me there's something external that has changed, but I have no clue as to what that might be -- as the process being started by my go tool will run just fine from a shell. And, it *does* run fine on my laptop (which granted is beefier, but again this server was running the tool just fine for years and I haven't done any big upgrades, but it *is* possible some minor underlying package update has severely changed the environment somehow). Unfortunately I don't have a system-wide 'snapshot' I can revert to.Perhaps this will end up being a question of Linux diagnostics more than Go but I haven't yet seen any way to tell *why* the process is being killed, whether it be due to some bug tickled by Go's exec or something else. The oom_reaper doesn't say a thing to system logs; I don't see my free RAM or swap suddenly drop.. I've even checked my server for rootkits out of paranoia :). Everything else on the system is just fine, I just cannot seem to run these scripts any more when launched from my go program (again, even a 'do-nothing' script that just sleeps a few times, then completes with exit status 0, no longer works -- it just gets 'killed').On Sat, Mar 2, 2024 at 7:39 PM Robert Engels <reng...@ix.netcom.com> wrote:Please clarify - does it work using the older versions of Go?On Mar 2, 2024, at 12:53 PM, Russtopia <rma...@gmail.com> wrote:I have tried rebuilding with go1.18.6, go1.15.15 with no difference.On Sat, Mar 2, 2024 at 6:23 PM Robert Engels <reng...@ix.netcom.com> wrote:I would  be also try reverting the Go version and ensure that it continues to work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor <i...@golang.org> wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia <rma...@gmail.com> wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.






-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/B6BF9BE3-1A33-4B05-92C2-1382B6F845CF%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

Please clarify - does it work using the older versions of Go?On Mar 2, 2024, at 12:53 PM, Russtopia  wrote:I have tried rebuilding with go1.18.6, go1.15.15 with no difference.On Sat, Mar 2, 2024 at 6:23 PM Robert Engels <reng...@ix.netcom.com> wrote:I would  be also try reverting the Go version and ensure that it continues to work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor <i...@golang.org> wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia <rma...@gmail.com> wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.





-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8E4E2802-8E23-4427-ACA1-DBA30EBD8B30%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

I would  be also try reverting the Go version and ensure that it continues to 
work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor  wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia  wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from 
>> Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B3E066FF-18F8-4BD9-A78B-2D98B930801B%40ix.netcom.com.

Re: [go-nuts] Limiting File Descriptors on Forked Linux cmd.exec

[Robert Engels]

The could be calling fork() as in the system call - which copies all file 
descriptors but I didn’t think Go processes could fork.

Seems you would need to remap stdin and stdout in the fork to do anything 
useful. 

This sounds very PHP - what goes around comes around. 

> On Mar 1, 2024, at 8:01 PM, Ian Lance Taylor  wrote:
> 
> On Fri, Mar 1, 2024 at 5:57 PM Jeff Stein  wrote:
>> 
>> I'm struggling to understand if I'm able to do something.
>> 
>> 
>> In my very odd use case we are writing a websever that handles connections 
>> via a forked process.
>> 
>> I have a listener process that listens for TCP connections.
>> 
>> So each net.Conn that comes in we pull off its file descriptor:
>> 
>> fd, err := conn.(*net.TCPConn).File()
>> 
>> duplicate that file descriptor and then fork off a process passing in that 
>> file descriptor.
>> 
>> In my forked handler I'll reconstruct the HTTP connection and "do stuff".
>> 
>> The concern I'm having is that it appears when I fork a process I inherit 
>> all of the parent file descriptors so if I have say 5 incoming connections 
>> and then I fork my child process technically could write to a different 
>> connection.
>> 
>> I've played around with the various options:
>> 
>> cmd.SysProcAttr = {Setpgid: false,}
>> 
>> and using cmd.ExtraFiles
>> 
>> No matter what I do I seem unable to limit the sub process to ONLY using the 
>> specific File Descriptor I want it to have access to.
>> 
>> I believe this is doable in C - but I'm not sure if I can do this in GoLang 
>> as-is without mods .
> 
> What you are describing shouldn't happen.  A child process should only
> get the file descriptors explicitly passed via the os/exec.Cmd fields
> Stdin, Stdout, Stderr, and ExtraFiles.  So tell us more: OS and
> version of Go, and what is showing you that all file descriptors are
> being passed down to the child.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUb27YBCyE52QisHLyB9XPPpEycMxt4FrFJogGsFMiemQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C7B700A5-F5BE-45D2-AA8B-E8A504627408%40ix.netcom.com.

Re: [go-nuts] Could we trade all the `ctx context.Context` arguments for one pointer in `g`?

[Robert Engels]

That is a good reason as to why putting timeouts and cancellation in the “context” always felt wrong to me. These are per request notions - and creating a new context to specify them seems off. But as I mentioned in the other post, without a concept of security context it doesn’t matter much - just throw anything and everything into the context and pass it around. On Feb 21, 2024, at 4:33 AM, 'Sean Liao' via golang-nuts  wrote:https://go.dev/issue/21335- seanOn Wed, Feb 21, 2024, 00:36 Sam Vilain  wrote:

  

  
  
Alright, well thanks for your input.
I do think these questions can be answered; exploring the use
  cases in a proposal format should hopefully show that the impact
  of closures would not normally be an issue.  Possibly the worst
  case is if you had a library to some external service, and at some
  low layer you're using a closure, well you might need to refactor
  that if you wanted to add cross–service tracing support.  But to
  be honest, it's probably better that the state you're depending on
  is captured in a struct instead of being scattered about loosely
  in a scope as with a closure.  And the common practice in Go is to
  return types that satisfy interfaces, not function references that
  you blind call.

I think I will submit a proposal, but please don't take that to
  imply that I think you're "wrong, or mistaken".  Your concerns are
  legitimate and the proposal should answer them cleanly.  In the
  proposal format, hopefully the "litigation", or more exploration
  of possible uses and abuses of the system, along with performance
  concerns, can be addressed.  I don't want to be dismissive of
  them, I just want to understand them.
I had a brief look on the Golang issues in Github and could not
  find any prior proposals along this line using "context" and
  "dynamic scope" as search terms, so I'll submit this as a "new"
  proposal for now.
Thanks again, and truly—thanks for responding, >100% better
  than people who just rolled eyes and marked thread as read.
Cheers,
  Sam

On 2/20/24 3:35 PM, Axel Wagner wrote:


  
  
If I may quote myself:


> And no matter which choice you make for the language -
  it means that if the programmers wanted the other, they'd have
  to jump through annoying hoops and get confusing and hard to
  debug problems.


Having a mechanism to get one or the other semantic doesn't
  change the fact that it's easy to choose wrongly by accident,
  as long as the effect is implicit. In fact, the mechanism you
  propose (AIUI) seems extra confusing: Having a function value
  sometimes create a new dynamic scope and sometimes not, is
  weird and seems like a recipe for frustration.



But really, convincing me isn't really the point, which is
  why I'm not super invested in litigating this (otherwise I
  might try to come up with realistic examples, for instance. Or
  explain further why I'm still not sure that this can be
  implemented efficiently). I'm just re-stating what, in the
  past, where the reasons why things like this have been
  rejected. In order to predict what I would consider a likely
  outcome of a proposal like this.


If you think I am wrong or misunderstanding you, you can
  always file a proposal to get a more official response.

  
  
  
On Tue, Feb 20, 2024 at
  8:18 PM Sam Vilain 
  wrote:


  
On 2/17/24 1:32 AM, Axel Wagner wrote:


  

  On Sat, Feb 17, 2024
at 2:09 AM Sam Vilain 
wrote:
  
  
I would argue that the matter can be simply
  decided by choosing the calling stack, not
  the destination stack.
  
   
  I agree that this is *one choice*. But the point
is, that *sometimes* you'd want one and *sometimes*
the other. And no matter which choice you make for
the language - it means that if the programmers
wanted the other, they'd have to jump through
annoying hoops and get confusing and hard to debug
problems. So if you want to justify either choice,
you have to make an argument that it is so
overwhelmingly more common what people would want,
that the cost of running into these problems is

Re: [go-nuts] Could we trade all the `ctx context.Context` arguments for one pointer in `g`?

[Robert Engels]

FWIW, I think having a bound context to an execution context is a valuable addition. One thing about Go that has always felt lacking is dynamic code loading execution. There is the plugin facility but it doesn’t seem to have been widely adopted. If it were, I think the Go team would find it needs a security model closer to that of Java - and being able to bind to the thread of execution and inherit there is critical to this working well. So I welcome your proposal. On Feb 20, 2024, at 6:37 PM, Sam Vilain  wrote:

  

  
  
Alright, well thanks for your input.
I do think these questions can be answered; exploring the use
  cases in a proposal format should hopefully show that the impact
  of closures would not normally be an issue.  Possibly the worst
  case is if you had a library to some external service, and at some
  low layer you're using a closure, well you might need to refactor
  that if you wanted to add cross–service tracing support.  But to
  be honest, it's probably better that the state you're depending on
  is captured in a struct instead of being scattered about loosely
  in a scope as with a closure.  And the common practice in Go is to
  return types that satisfy interfaces, not function references that
  you blind call.

I think I will submit a proposal, but please don't take that to
  imply that I think you're "wrong, or mistaken".  Your concerns are
  legitimate and the proposal should answer them cleanly.  In the
  proposal format, hopefully the "litigation", or more exploration
  of possible uses and abuses of the system, along with performance
  concerns, can be addressed.  I don't want to be dismissive of
  them, I just want to understand them.
I had a brief look on the Golang issues in Github and could not
  find any prior proposals along this line using "context" and
  "dynamic scope" as search terms, so I'll submit this as a "new"
  proposal for now.
Thanks again, and truly—thanks for responding, >100% better
  than people who just rolled eyes and marked thread as read.
Cheers,
  Sam

On 2/20/24 3:35 PM, Axel Wagner wrote:


  
  
If I may quote myself:


> And no matter which choice you make for the language -
  it means that if the programmers wanted the other, they'd have
  to jump through annoying hoops and get confusing and hard to
  debug problems.


Having a mechanism to get one or the other semantic doesn't
  change the fact that it's easy to choose wrongly by accident,
  as long as the effect is implicit. In fact, the mechanism you
  propose (AIUI) seems extra confusing: Having a function value
  sometimes create a new dynamic scope and sometimes not, is
  weird and seems like a recipe for frustration.



But really, convincing me isn't really the point, which is
  why I'm not super invested in litigating this (otherwise I
  might try to come up with realistic examples, for instance. Or
  explain further why I'm still not sure that this can be
  implemented efficiently). I'm just re-stating what, in the
  past, where the reasons why things like this have been
  rejected. In order to predict what I would consider a likely
  outcome of a proposal like this.


If you think I am wrong or misunderstanding you, you can
  always file a proposal to get a more official response.

  
  
  
On Tue, Feb 20, 2024 at
  8:18 PM Sam Vilain 
  wrote:


  
On 2/17/24 1:32 AM, Axel Wagner wrote:


  

  On Sat, Feb 17, 2024
at 2:09 AM Sam Vilain 
wrote:
  
  
I would argue that the matter can be simply
  decided by choosing the calling stack, not
  the destination stack.
  
   
  I agree that this is *one choice*. But the point
is, that *sometimes* you'd want one and *sometimes*
the other. And no matter which choice you make for
the language - it means that if the programmers
wanted the other, they'd have to jump through
annoying hoops and get confusing and hard to debug
problems. So if you want to justify either choice,
you have to make an argument that it is so
overwhelmingly more common what people would want,
that the cost of running into these problems is

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

If you implemented it and it works - I guess I don’t understand your question. You must know how tcp/http works. On Feb 11, 2024, at 8:25 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:Yes I did implement. You can check it here.https://github.com/roychowdhuryrohit-dev/slugOn Sunday, February 11, 2024 at 5:00:02 PM UTC-8 Robert Engels wrote:If it is a web server that supports http clients you still have to implement the protocol correctly. At the lowest level that is what all web server implementations do - they read from the socket directly. On Feb 11, 2024, at 7:54 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:As stated earlier, objective is to create a web server without using net/http, instead directly reading from the socket connection.On Sunday, February 11, 2024 at 4:49:09 PM UTC-8 Robert Engels wrote:If you have http keep alive on - either side should block when reading - it is a full duplex connection. You will only get eof if the tcp connection is closed. If http keep alive is off, then the connection is closed after the server sends the response - the tcp protocol allows the client to read this data even though the connection is shut down. On Feb 11, 2024, at 5:36 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:Yes I got it but I want to know if reader.ReadString("\n") is supposed to throw io.EOF rather than blocking for next request in the connection.On Sunday, February 11, 2024 at 2:30:44 PM UTC-8 Robert Engels wrote:There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if it exists in the pool.



-- 
You received this message because you are subscribed to the G

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

If it is a web server that supports http clients you still have to implement the protocol correctly. At the lowest level that is what all web server implementations do - they read from the socket directly. On Feb 11, 2024, at 7:54 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:As stated earlier, objective is to create a web server without using net/http, instead directly reading from the socket connection.On Sunday, February 11, 2024 at 4:49:09 PM UTC-8 Robert Engels wrote:If you have http keep alive on - either side should block when reading - it is a full duplex connection. You will only get eof if the tcp connection is closed. If http keep alive is off, then the connection is closed after the server sends the response - the tcp protocol allows the client to read this data even though the connection is shut down. On Feb 11, 2024, at 5:36 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:Yes I got it but I want to know if reader.ReadString("\n") is supposed to throw io.EOF rather than blocking for next request in the connection.On Sunday, February 11, 2024 at 2:30:44 PM UTC-8 Robert Engels wrote:There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if it exists in the pool.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/919a5a2d-bd99-4f9c-b9fd-cfa2bd0a3862n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Go

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

If you have http keep alive on - either side should block when reading - it is a full duplex connection. You will only get eof if the tcp connection is closed. If http keep alive is off, then the connection is closed after the server sends the response - the tcp protocol allows the client to read this data even though the connection is shut down. On Feb 11, 2024, at 5:36 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:Yes I got it but I want to know if reader.ReadString("\n") is supposed to throw io.EOF rather than blocking for next request in the connection.On Sunday, February 11, 2024 at 2:30:44 PM UTC-8 Robert Engels wrote:There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if it exists in the pool.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/919a5a2d-bd99-4f9c-b9fd-cfa2bd0a3862n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8014b001-ef32-4937-b31e-1ce6b24baf48n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nut

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if it exists in the pool.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/919a5a2d-bd99-4f9c-b9fd-cfa2bd0a3862n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0033731F-373E-43B2-9348-11506C797D11%40ix.netcom.com.

Re: [go-nuts] help with thread limit

[Robert Engels]

You can use cpuctrl or similar to limit the number of visible cores when you start the process. On Jan 31, 2024, at 8:43 PM, Steve Roth  wrote:I am running Go code on a shared web hosting server from a major hosting company.  Using cgroups, they limit the number of threads any user can create to 25.  Up until a week ago, they had me running on a server with 24 cores, and everything worked fine.  Now they've moved me to a new server with 128 cores.  The Go runtime thinks it should be able to create that many threads, and it can't, and all of my Go programs crash with, "runtime: failed to create new OS thread".Setting GOMAXPROCS doesn't help this, since it only controls the number of active threads, not the total number of threads.  The Go runtime still thinks it can create as many threads as there are cores, and crashes the program when it's blocked from doing so.Is there any way around this?  Or is Go just completely unusable in an environment where the per-user process limit is less than the number of cores?Many thanks,Steve



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAnpqKHvfni_U%2BkCHp0hhUryDytPzQ%2BSqHZTcAFH9HNXgRCy_A%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/51EA8D38-5882-4F9B-A660-6E47627C5846%40ix.netcom.com.

Re: [go-nuts] Re: Good examples of Go back ends?

[Robert Engels]

github.com/robaho/go-trader has many easy to follow examples - authentication, multicast , routing, rest apis, websockets. A major system not covered though is db access. On Jan 22, 2024, at 9:23 AM, george looshch  wrote:hi Jason,thanks a million for pointing out the vagueness of my question! English isn’t my mother tongue so now i see where you’re coming fromwhat i meant was examples of real-world web server with routing, authentication, DB, etc.curated lists have libraries and frameworks, what i’m looking for is examples of usages these libraries and frameworks in production. Search on github didn’t yield any good results, unfortunatelyOn Monday, January 22, 2024 at 2:13:17 PM UTC Jason E. Aten wrote:This question is too vague.You are likely to get more helpful answers if you specify what kind of "backend" you are looking for.  As it is, we can only guess.  Do you want backends that are web servers? (see the standard library net/http or the caddy web server)  Is it a backend for iOS iPhone Apps? For Android Apps? That respond to a specific kind of RPC such as gRPC? That simply access a database?...  A relational database? A non-relational database (graph?, vector?, full-text search?) Pocketbase is a backend mentioned recently on hackernews, that is written in Go and seems to do alot.  Perhaps it is similar to firebase, just going by the name. I have not used it myself.  I cannot say whether it is a "good example" or not, because I've not used it, and similarly this is too vague a criteria (good at what?)https://github.com/pocketbase/pocketbaseGenerally, go over to github and search for the kind of backend you want, and select those projects that are written in Go on the left side filter click-boxes.  You could also look at the curated lists of Go libraries such as https://awesome-go.com/On Sunday, January 21, 2024 at 4:57:42 PM UTC+1 george looshch wrote:hi!can i please ask if someone knows good examples of back ends written in Go? If not good, just production code would be great as well!thanks in advance and have a great rest of the weekend!



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/bcb429bb-4cfb-4421-be4e-b1ffbcd5e228n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1A0098A5-D948-4C46-8E93-6AB01D85AA39%40ix.netcom.com.

Re: [go-nuts] Re: does anyone have an example of a Go kernel module

[Robert Engels]

Hasn’t been touched in 6 years and it doesn’t work. I guess they figured out it wasn’t worth the effort. The only plausible way to write a kernel module in Go is… don’t.Maybe run a user space Go process and some sort of shared memory communication. On Jan 7, 2024, at 8:06 PM, Hữu Hà  wrote:Maybe you mean something like this ?https://github.com/gopher-os/gopher-osVào lúc 04:22:10 UTC+7 ngày Chủ Nhật, 7 tháng 1, 2024, Siglo XIX đã viết:I have tried many ways but now that the ecosystem is more mature maybe someone knows of an example of how to make a Linux kernle module with Go,.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cf1a03df-6782-4ef7-92af-4d3f3dc833c8n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/28E9FBFC-4649-4FB3-8D50-5439FF5935EB%40ix.netcom.com.

Re: [go-nuts] How to convert a net.Addr into a netip.AddrPort ?

[Robert Engels]

github.com/robaho/go-trader has udp multicast support with both server and client codeOn Dec 24, 2023, at 11:05 AM, christoph...@gmail.com  wrote:Hello,I'm developping a UDP client server program. Everything is clear on the server side. I'm just a bit confused with the client side. In C we can specify the network address and set the port to 0 so that a free port is picked. Apparently we can't do that with the net API. The client must use net.DialUDP() to get a connection it can use to send messages to the remote server. If the laddr is set to nil, net.DialUDP will pick the local address and port. We thus can't pick the local address and let the system pick the port. It'll pick both. I assume it is to solve the IPv4 and IPv6 coexistence. The returned net.UDPAddr has a LocalAddress() method returning a net.Addr. How can I convert this net.Addr into a netip.AddrPort ?  I can only get a string version of the address. Are we expected to use the ResolveUDP to convert the string into the UDPAddr ?  



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f248a95d-ac11-4e9f-81d8-d53d047d24ffn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7549AFA2-126E-4B7C-87A7-03B47F916368%40ix.netcom.com.

Re: [go-nuts] Mutex contention in sql.Close()

[robert engels]

It is contending with sql.rows.awaitDone().

My guess is that when you close rows it has to wait for db acknowledgement - 
that this is a false positive based on wall time and not cpu time - that the 
mutex is waiting for the cleanup operation to complete.

> On Dec 13, 2023, at 1:25 PM, Joshua Garnett  wrote:
> 
> Hi All,
> 
> Has anyone else observed significant mutex contention in sql.Close() at 
> scale?  I haven't had much success working around it beyond spinning up 
> additional nodes.
> 
> App details
> Go 1.21.4
> debian:bookworm container
> App is running on an instance with 18 physical cores (36 vCPU) and 72GB of 
> memory (only main process running)
> Connecting to PostgreSQL 15.3 AWS Aurora
> MaxOpenConns set to 125
> Here is a screenshot of the pprof:
> 
> 
> 
> Thanks in advance!
> 
> --Josh
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJAXYdi2A-rJ9YhbJJB0a1TE_MbrHL2rMOSKP1Rb%3DxHc9dOveQ%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0A3D8BF0-0062-40D7-B84E-E9397F935EB2%40ix.netcom.com.

Re: [go-nuts] detecting deleted file that is still open and appending without error in Go?

[Robert Engels]

As the links pointed out you can no longer relink but you can certainly access the data. So the append it working - you just can’t see it using ls. Anyway, I guess you have enough knowledge now to address. On Dec 10, 2023, at 1:34 PM, Jason E. Aten  wrote:Thanks Robert.On Sunday, December 10, 2023 at 6:30:11 PM UTC Robert Engels wrote:Not sure how you are detecting that the append it not working - I noticed that the binary log was not growing, and its update timestamp was not changing. I have an alias, lh, that is





alias lh='ls -alFtrh|tail'that I run regularly in the course of work to see what has changed last in a directory. It is very useful, if you do not use something like it.Only by luck did I happen to notice that the file was not being grown, which concerned me and so I investigated via/proc/pid/fd, saw the deletion and then looked at the git log for the file. The updates stopped hitting the file (because the updated time on the file stopped changing) right after I added it to git.On Sunday, December 10, 2023 at 6:30:11 PM UTC Robert Engels wrote:> it usually does because a client can relink the inode to another file name using a syscallInteresting. What syscalls relink it? Ah... https://serverfault.com/questions/168909/relinking-a-deleted-file seems to suggest that it used to be possible, but not after 2011 / linux kernel 2.6.39 because of security concerns.> I, of course, did test this before posting my solution and at that time it actually worked for me. What I wasn't aware of is that it only worked on tmpfs filesystems but not on e.g. ext3. Furthermore this feature got completely disabled in 2.6.39, see the commit. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=aae8a97d3ec30788790d1720b71d76fd8eb44b73 ) So therefore this solution won't work with kernel 2.6.39 or newer anymore and in earlier versions it depends on the filesystem. – tnimeu Jan 26, 2012 at 14:04where the link is to a commit:  "fs: Don't allow to create hardlink for deleted file" from 2011.Anyway, the Stat approach using the original filepath seems to be working: I can recognize the file shrinking or not growing when it should, and re-create it from memory.Thanks All.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e322acad-3070-4b51-ac23-c541b830c0aen%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/127C2A77-3888-47EC-936D-C4B6B93BA44B%40ix.netcom.com.

Re: [go-nuts] detecting deleted file that is still open and appending without error in Go?

[Robert Engels]

Not sure how you are detecting that the append it not working - it usually does 
because a client can relink the inode to another file name using a syscall 

I think what you want to do it either 

1. Open the file exclusively so git cannot do this. 
2. Check that the file create time is the same as when you first opened the 
file (you have to make the fstat() by name)

> On Dec 10, 2023, at 10:59 AM, Jan Mercl <0xj...@gmail.com> wrote:
> 
> On Sun, Dec 10, 2023 at 5:41 PM Jason E. Aten  wrote:
> 
>> My question is: is there a way to have the Go process detect if the file it 
>> is writing to has been deleted by another process (git in this case) so that 
>> attempting to append to the file is no longer effective?
> 
> It is effective and [most] operations on the file continue to work as
> usual. "Removing" a file, like in `os.Remove` is just removing one
> reference to it. Only when the reference count drops to zero is the
> file deleted.
> 
> This is, AFAIK, how nixes work and it's IMO actually a neat feature.
> It enables, for example, updating files without disrupting processes
> that have those files opened. Less advanced operating systems, to
> achieve the same effect, have to reboot etc.
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAA40n-VZDOdwMOqqmAc-RgpGyNkOi0LvyFR%2BKbzem4PqNNwrYQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2B61F3AA-7C01-4172-A7B2-684CDFADC5A9%40ix.netcom.com.

Re: [go-nuts] Re: Is "When in doubt, use a pointer receiver" misleading advice?

[Robert Engels]

What I’m suggesting is that imagine a dev changes that code and has version() 
access the request property…

This is why if you are sharing data in a concurrent way you need to be very 
careful of all usages. 

The safest solution is to use immutable objects - of which the non pointer are. 

So saying - just use pointer receivers - woefully underestimates the work to do 
shared concurrent code correctly. 

> On Nov 15, 2023, at 10:33 AM, Marvin Renich  wrote:
> 
> * Robert Engels  [231114 21:55]:
>>   Switching to pointer receivers everywhere actually makes this worse. Any
>>   access is potentially a data race. 
>>   It stills seems like this is a compiler issue. There needs to be a way to
>>   synchronize the pointer to value copy in conjunction with other
>>   synchronization. 
>>   The only way to do this would be to write your own pointer to value
>>   conversion methods that have synchronization. 
> 
> I don't understand what you are saying.  Are you saying that if you take
> the code from Dave Cheney's article, and change the receiver for the
> version method from value to pointer:
> 
>  func (*RPC) version() int {
> 
> that there is still a race in his code?  Or are you saying that with
> that change you could add code where the act of calling a method is
> inherently a race?  If the latter, can you give an example?
> 
> ...Marvin
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/ZVTyykWUgiwPLH9n%40basil.wdw.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A1302C26-7DED-483C-A250-FB84D92E0661%40ix.netcom.com.

Re: [go-nuts] Re: Is "When in doubt, use a pointer receiver" misleading advice?

[Robert Engels]

Switching to pointer receivers everywhere actually makes this worse. Any access is potentially a data race. It stills seems like this is a compiler issue. There needs to be a way to synchronize the pointer to value copy in conjunction with other synchronization. The only way to do this would be to write your own pointer to value conversion methods that have synchronization. On Nov 14, 2023, at 8:36 PM, Mike Schinkel  wrote:On Tuesday, November 14, 2023 at 6:16:58 PM UTC-5 burak serdar wrote:It is a data race because calling rpc.version() makes a copy of rpc,which causes reading the field rpc.result concurrently while it isbeing written by the goroutine. Thank you for explaining. I think I am starting to see it.On Tuesday, November 14, 2023 at 7:08:57 PM UTC-5 burak serdar wrote:I do not agree that this is because how the compiler works. A valuereceiver is equivalent to pass-by-value argument, that is:rcp.version()is equivalent to:RPC.version(rpc)thus, creating the copy of the rpc variable. So, the compiler maychoose to avoid the race by not copying it, or by inlining the versionfunction, but according to the spec, it is passed by value, i.e., itis copied.I agree with your assertion — if you fix it to work correctly by using RPC.version(*rpc)— but that is actually a different concern than I and I think Robert Engels were discussing.  Your example above calls a method directly on the struct, we were discussing calling via an interface. Given that Go already performs some "magic" when it calls an interface — i.e. when the interface value contains a pointer and the method receive is a value — I think he was exploring whether it would be viable to avoid creating a copy when the method does not read or update any property.Still, that doesn't seem to be all that useful because most of the time a value method would want to read a property so it is probably not worth the effort.But back to the original concern. It seems that race conditions do not arise when using mixed receivers while only executing the main goroutine, right?And if using only pointer receivers then we can still have a data race if those methods are being called in multiple goroutines where methods both read and update the same properties, right?.  Given that, the race conditions and the mixed pointers seem orthogonal concerns?   But I do now recognize that if you are using the same objects across goroutines and you have an updating pointer method then calling any value method would create a race condition so mixing receivers is more likely to cause a race condition when sharing objects across goroutines.Still, it seems to me then that the guidance should be to not to access any object concurrently in more than one goroutine instead of to avoid mixed receivers, since that laters only addresses part of the race condition problem?  (BTW, by "not-concurrently" I mean that you could first access in goroutine 1 then pass it via a channel to goroutine 2 where you could access it, but once goroutine 1 submits the object to the channel it should no longer attempt to access it.)Or again, am I missing something obvious?   -Mike



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8f46fecc-1d5f-4507-8209-76adef294bafn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/F4895A80-66A0-4957-B369-0CDFD6DB16EA%40ix.netcom.com.

Re: [go-nuts] Re: Is "When in doubt, use a pointer receiver" misleading advice?

[Robert Engels]

Lastly, the only reason that changing to a pointer receiver “solves” the race 
is because the field in the object isn’t accessed - something the compiler 
could detect in the other version as well. 

> On Nov 13, 2023, at 10:50 PM, Robert Engels  wrote:
> 
> Similarly, even in a single threaded Java program objects are copied/moved 
> behind the scenes - these don’t generate race conditions as the runtime 
> ensures there isn’t one.
> 
>> On Nov 13, 2023, at 10:47 PM, Robert Engels  wrote:
>> 
>> To me this is a limitation of the compiler. If a passed argument is unused 
>> it doesn’t even need to be created. Similarly if only a few fields are used 
>> in a called function a sparse data object can be sent. It only becomes a 
>> race if the data being copied is actually used - then the race is also 
>> obvious.
>> 
>>>>> On Nov 13, 2023, at 10:27 PM, 'Dan Kortschak' via golang-nuts 
>>>>>  wrote:
>>>>> 
>>>>> On Mon, 2023-11-13 at 19:38 -0800, Mike Schinkel wrote:
>>>>> I have been wondering for a while why the advice against mixing
>>>>> pointer and value receivers, which GoLang so often flags me for
>>>>> doing.
>>> 
>>> https://dave.cheney.net/2015/11/18/wednesday-pop-quiz-spot-the-race
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/3cc8112142c151ffea7fe640a2e7150fafcf202d.camel%40kortschak.io.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/32B1BD5C-56E8-4518-A470-3A0A9305B25A%40ix.netcom.com.

Re: [go-nuts] Re: Is "When in doubt, use a pointer receiver" misleading advice?

[Robert Engels]

Similarly, even in a single threaded Java program objects are copied/moved 
behind the scenes - these don’t generate race conditions as the runtime ensures 
there isn’t one. 

> On Nov 13, 2023, at 10:47 PM, Robert Engels  wrote:
> 
> To me this is a limitation of the compiler. If a passed argument is unused 
> it doesn’t even need to be created. Similarly if only a few fields are used 
> in a called function a sparse data object can be sent. It only becomes a race 
> if the data being copied is actually used - then the race is also obvious.
> 
>>> On Nov 13, 2023, at 10:27 PM, 'Dan Kortschak' via golang-nuts 
>>>  wrote:
>>> 
>>> On Mon, 2023-11-13 at 19:38 -0800, Mike Schinkel wrote:
>>> I have been wondering for a while why the advice against mixing
>>> pointer and value receivers, which GoLang so often flags me for
>>> doing.
>> 
>> https://dave.cheney.net/2015/11/18/wednesday-pop-quiz-spot-the-race
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/3cc8112142c151ffea7fe640a2e7150fafcf202d.camel%40kortschak.io.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C5B3871F-B3EE-4360-BD3A-E3E77A35BA8E%40ix.netcom.com.

Re: [go-nuts] Re: Is "When in doubt, use a pointer receiver" misleading advice?

[Robert Engels]

To me this is a limitation of the compiler. If a passed argument is unused it 
doesn’t even need to be created. Similarly if only a few fields are used in a 
called function a sparse data object can be sent. It only becomes a race if the 
data being copied is actually used - then the race is also obvious. 

> On Nov 13, 2023, at 10:27 PM, 'Dan Kortschak' via golang-nuts 
>  wrote:
> 
> On Mon, 2023-11-13 at 19:38 -0800, Mike Schinkel wrote:
>> I have been wondering for a while why the advice against mixing
>> pointer and value receivers, which GoLang so often flags me for
>> doing.
> 
> https://dave.cheney.net/2015/11/18/wednesday-pop-quiz-spot-the-race
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3cc8112142c151ffea7fe640a2e7150fafcf202d.camel%40kortschak.io.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C2C059F7-3431-41B5-B77A-59C89879AD1D%40ix.netcom.com.

Re: [go-nuts] Writing bitmap data to a canvas with WebAssembly

[robert engels]

Your numbers do not make sense.

At 300 us a frame, it would be more than 3k frames a second - which would only 
be possible for an extremely tiny area.

You need to include the time to produce the data when considering “render time” 
- otherwise you are talking apples and oranges.

> On Nov 7, 2023, at 9:43 PM, Kai O'Reilly  wrote:
> 
> For future reference for anyone who comes across this thread, you can 
> directly pass RGBA image data to a canvas, which is around 200 times faster 
> than encoding it to a PNG and rendering it to an offscreen image. Also, 
> passing the unsafe pointer to the slice to JavaScript instead of copying the 
> bytes improves performance by another factor of 5. These two changes combined 
> reduced the render time for me from roughly 300 milliseconds to 300 
> microseconds (a 1000x improvement). This performance is enough to run an 
> interactive pure Go GUI through WASM without any noticeable lag, just by 
> writing images to a canvas.
> 
> This is the relevant Go code:
> 
> ```go
>  sz := dw.image.Bounds().Size()
> ptr := uintptr(unsafe.Pointer([0]))
> js.Global().Call("displayImage", ptr, len(dw.image.Pix), sz.X, sz.Y)
> ```
> 
> And this is the relevant JS code:
> 
> ```js
> const appCanvas = document.getElementById('app');
> const appCanvasCtx = appCanvas.getContext('2d');
> 
> let wasm;
> let memoryBytes;
> 
> // displayImage takes the pointer to the target image in the wasm linear 
> memory
> // and its length. Then, it gets the resulting byte slice and creates an 
> image data
> // with the given width and height.
> function displayImage(pointer, length, w, h) {
>   // if it doesn't exist or is detached, we have to make it
>   if (!memoryBytes || memoryBytes.byteLength === 0) {
> memoryBytes = new Uint8ClampedArray(wasm.instance.exports.mem.buffer);
>   }
> 
>   // using subarray instead of slice gives a 5x performance improvement due 
> to no copying
>   let bytes = memoryBytes.subarray(pointer, pointer + length);
>   let data = new ImageData(bytes, w, h);
>   appCanvasCtx.putImageData(data, 0, 0);
> }
> ```
> In the JS code, `wasm` is initialized as the result of 
> `WebAssembly.instantiateStreaming`.
> 
> On Friday, July 24, 2020 at 10:53:14 AM UTC-7 Mark Farnan wrote:
> Little old, but this might also help. 
> 
>  A while back I built a helper package to deal with these issues for canvas 
> rendering from Go.  
> 
> https://github.com/markfarnan/go-canvas 
>   
> 
> I'm currently working on it to add  WebGL support & get it working in TinyGo 
> (some issues still).
> 
> Regards
> 
> Mark.
> 
> 
> 
> On Sunday, 22 March 2020 at 09:08:40 UTC+1 Scott Pakin wrote:
> I figure I ought to follow up with some results.  First, I got the suggested 
> approach of local render + js.CopyBytesToJS 
>  + update canvas from image 
> to work, so thanks, Agniva and Howard!  Second, for the benefit of future 
> readers of this thread, one thing that wasn't obvious to me is that one needs 
> to render the image data in a browser-recognizable image format—I used PNG 
> —not raw {red, 
> green, blue, alpha} bytes as is needed when writing directly to a canvas 
> 's image data.  Third, I used 
> JavaScript code like the following to update an invisible img 
>  then copy the image data from 
> there to a visible canvas:
> 
> function copyBytesToCanvas(data) {
> let blob = new Blob([data], {"type": "image/png"});
> let img = document.getElementById("myImage");
> img.onload = function() {
> let canvas = document.getElementById("myCanvas");
> let ctx = canvas.getContext("2d");
> ctx.drawImage(this, 0, 0);
> };
> img.src = URL.createObjectURL(blob);
> }
> 
> Fourth, the performance is indeed substantially faster than my previous 
> approach based on using SetIndex 
>  to write directly to the 
> canvas, even though the new approach requires the extra steps of encoding the 
> image in PNG format and copying the image data from an img to a canvas.  The 
> following performance data, measured with Go 1.14 and Chromium 80.0.3987.132 
> on an Ubuntu Linux system, is averaged over 10 runs:
> 
> Old: 686.9 ± 7.6 ms
> New: 290.4 ± 4.1 ms (284.3 ± 4.2 on the WebAssembly side plus 6.0 ± 2.3 on 
> the JavaScript side)
> 
> This is the time to render a simple 800×800 gradient pattern.
> 
> I hope others find this useful.
> 
> — Scott
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> 

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Robert Engels]

Better to rewrite the native portions in Go. 

If you’re not going to do that, just give instructions on how to build and let 
the app builder figure out how to package it. 

> On Nov 3, 2023, at 1:47 PM, Jason E. Aten  wrote:
> 
> 
> It is still rough around lots of edges, but you are wanting to write your own 
> deployment system, you might find in https://nixos.org/ some inspiration.
> 
> It generalizes the idea of cryptographic hashed based dependency management 
> to, well, everything.
> 
> Sadly not that easy to use; it may be harsh to inflict on end users.  But it 
> tackles the reproducibility issue (it works on my machine! but not yours?) 
> head on.
> You might profitably write some glue between Go and Nix that provides a nice 
> end user experience.
> 
>> On Friday, November 3, 2023 at 5:23:21 PM UTC Jan wrote:
>> That works if what I'm doing is a end product (a server or something). But 
>> what I'm doing is in itself a library. Imagine if every library offers a 
>> docker/VM image, how is the end-user supposed to merge those docker images ? 
>> The docker is also not a viable/ergonomic solution in my case. I mean ... 
>> actually I do offer a docker for one of my projects, along with Jupyter, a 
>> Tutorial and demo. But generically, it is a library, and I would like it to 
>> be easily `go get`able. Or at most with one simple/standard step.
>> 
>> Yes, I hear you with respect to unnecessary extra dependencies (X11) -- at 
>> least make them optional, right ? But In my case the dependencies are 
>> essential, and the docker doesn't really solve the problem...  But also I'm 
>> running out of hope that there would be anything to solve it, I'm almost 
>> thinking I should write something myself.
>> 
>> 
>>> On Thursday, November 2, 2023 at 11:54:01 PM UTC+1 Jason E. Aten wrote:
>>> What I would do for that case would be to deliver the users a either a VM 
>>> image (heavy), or a docker container (lighter) with batteries included 
>>> inside. 
>>> 
>>> There is often little need to write a docker file unless you really want. 
>>> Just get everything working inside docker and "docker commit" it to an 
>>> image. I do this often for projects that need python/R + 100s of libraries. 
>>>  Its really the only sane way to deliver sprawling dependencies that assume 
>>> they can write all over the filesystem.  
>>> 
>>> One hint: I recently did this for code that wanted to output graphics, and 
>>> that needed X11 (or Xvfb), and that wanted systemd, which is available in 
>>> docker but used to be discouraged so is not available in many base images. 
>>> So I would recommend starting an docker image that already supports systemd 
>>> and X11, if graphics (or x11vnc, say) is ever going to be desired. Its a 
>>> pain to add after the fact. Podman claims to be useful for this, but I 
>>> found in immature and not really production ready when trying to run it on 
>>> blah standard Ubuntu 22.
>>> 
 On Wednesday, November 1, 2023 at 9:08:51 PM UTC Jan wrote:
 Thanks @Jason, but the point was exactly not need to do anything extra: no 
 manual unzipping of a file, or manual Makefile/Magefile.
 
 Let's say project in Go links some 30 external modules, 5 of them have 
 their own specific steps that need to be read, understood and and run to 
 install them ... very quickly the user just gives up from 
 installing/compiling the thing, with good reason ... 
 
 The Go toolset solves that beautifully for Go only projects. But at least 
 in my line of work, that's not feasible, I need to interface with 
 libraries that in turn require other libraries in C++ (OpenXLA/llvm for 
 JIT), Rust (HuggingFace Tokenizer), C (Gtk), etc.
 
 cheers
 
 
 
 
> On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:
> > including the `.a` (static libraries) in the Github just got to its 
> > limit (100Mb)
> 
> I have used bzip2 to compress libraries that are too big for Github. If 
> that gets them under the limit, then great. 
> Just provide installation instructions or a Makefile target that 
> decompresses them once checked out.
> 
> (Or xz? Apparently it can compress more than bzip2; but I have no 
> experience with it. https://en.wikipedia.org/wiki/XZ_Utils )
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/149f28a5-417e-4642-9c04-13f0254e2dcbn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web 

Re: [go-nuts] Re: database/sql: Latency when calling DB.QueryContext()

[Robert Engels]

The internet. The gift that keeps on giving. 

> On Nov 3, 2023, at 7:54 AM, Al Corona  wrote:
> 
> Off topic question. But does anyone know how I can gain admin access on my 
> work computer? 
> 
>> On Thursday, June 29, 2017 at 12:01:16 AM UTC-7 Kai Zhang wrote:
>> Go version 1.8.3
>> 
>> In DB.QueryContext(), Go will close long live connection which exceeded the 
>> duration setted by  DB.SetConnMaxLifetime(), the code is below
>> database/sql/sql.go
>>  895 func (db *DB) conn(ctx context.Context, strategy connReuseStrategy) 
>> (*driverConn, error) {
>>  896 db.mu.Lock()
>>  897 if db.closed {
>>  898 db.mu.Unlock()
>>  899 return nil, errDBClosed
>>  900 }
>>  901 // Check if the context is expired.
>>  902 select {
>>  903 default:
>>  904 case <-ctx.Done():
>>  905 db.mu.Unlock()
>>  906 return nil, ctx.Err()
>>  907 }
>>  908 lifetime := db.maxLifetime
>>  909 
>>  910 // Prefer a free connection, if possible.
>>  911 numFree := len(db.freeConn)
>>  912 if strategy == cachedOrNewConn && numFree > 0 {
>>  913 conn := db.freeConn[0]
>>  914 copy(db.freeConn, db.freeConn[1:])
>>  915 db.freeConn = db.freeConn[:numFree-1]
>>  916 conn.inUse = true
>>  917 db.mu.Unlock()
>>  918 if conn.expired(lifetime) {
>>  919 conn.Close()
>>  920 return nil, driver.ErrBadConn
>>  921 }   
>>  
>>
>>  922 return conn, nil
>>  923 }
>> 
>> 
>> The problem is if driver.Close() has some handshake to tear down the 
>> connection, this may introduce unexpected latency. But the driver.Conn 
>> interface not mention this.
>> database/sql/driver/driver.go
>>  143 type Conn interface {
>>  144 // Prepare returns a prepared statement, bound to this connection.
>>  145 Prepare(query string) (Stmt, error)
>>  146 
>>  147 // Close invalidates and potentially stops any current
>>  148 // prepared statements and transactions, marking this
>>  149 // connection as no longer in use.
>>  150 //
>>  151 // Because the sql package maintains a free pool of
>>  152 // connections and only calls Close when there's a surplus of
>>  153 // idle connections, it shouldn't be necessary for drivers to
>>  154 // do their own connection caching. 
>>  
>>
>>  155 Close() error
>>  156 
>>  157 // Begin starts and returns a new transaction.
>>  158 //
>>  159 // Deprecated: Drivers should implement ConnBeginTx instead (or 
>> additionally).
>>  160 Begin() (Tx, error)
>>  161 }
>> 
>> There is already a goroutine to clean the connections, why still need a 
>> synchronous check? 
>> database/sql/sql.go
>>  750 func (db *DB) connectionCleaner(d time.Duration) {  
>>  
>>
>>  751 const minInterval = time.Second
>>  752 
>>  753 if d < minInterval {
>>  754 d = minInterval
>>  755 }
>>  756 t := time.NewTimer(d)
>>  757 
>>  758 for {
>>  759 select {
>>  760 case <-t.C:
>>  761 case <-db.cleanerCh: // maxLifetime was changed or db was 
>> closed.
>>  762 }
>>  763 
>>  764 db.mu.Lock()
>>  765 d = db.maxLifetime
>>  766 if db.closed || db.numOpen == 0 || d <= 0 {
>>  767 db.cleanerCh = nil
>>  768 db.mu.Unlock()
>>  769 return
>>  770 }
>>  771 
>>  772 expiredSince := nowFunc().Add(-d)
>>  773 var closing []*driverConn
>>  774 for i := 0; i < len(db.freeConn); i++ {
>>  775 c := db.freeConn[i]
>>  776 if c.createdAt.Before(expiredSince) {
>>  777 closing = append(closing, c)
>>  778 last := len(db.freeConn) - 1
>>  779 db.freeConn[i] = db.freeConn[last]
>>  780 db.freeConn[last] = nil
>>  781 db.freeConn = db.freeConn[:last]
>>  782 i--
>>  783 }
>>  784 }
>> 
>> 
>> 
>> Is this by design or need to be fixed? 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/c5f07284-32a9-4bc9-954c-edc7a8913843n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 

Re: [go-nuts] Suggestions on optimizing Go GC

[Robert Engels]

A few hundred milliseconds is pretty short. Are you certain you don’t have a 
memory leak?

Still, just run N request handler PROCESSES each with 1/N the heap you have 
allocated now. 

If each of these continue to grow to unmanageable size - simply kill and spawn 
a new request process - with a load balancer to hit the available ones for any 
request. 

You are going to spend extra cpu on GC overall - but you can limit the GC in 
each process since you’re going to kill it once it reaches a certain heap size. 

If that won’t work then there is really only one choice - the off heap memory 
that others have mentioned. 

You may have an an allocation pattern that needs a generational collector which 
Go doesn’t offer. 

> On Oct 31, 2023, at 12:47 AM, Zhihui Jiang  wrote:
> 
> 
> 
> On Monday, October 30, 2023 at 10:12:08 PM UTC-7 Robert Engels wrote:
> What is the average wall time of a request?
> The average latency is a few hundred milliseconds.  
> 
> Based on what you wrote it appears that handling a single request generates a 
> lot of garbage - high allocation rate - and for this to be significant I 
> suspect the runtime is also significant - which implies to me a spawn and 
> destroy request handler is your best bet. 
> I actually didn't quite get your suggestion earlier. We are using gRPC and I 
> think for each request we already have separate goroutines to handle it. Can 
> you explain a little bit more about spawn and destroy request handler? 
> 
>> On Oct 30, 2023, at 11:56 PM, Zhihui Jiang  wrote:
>> 
>> Hi Michael, Jason and Robert, thanks a lot for the replies and suggestions!
> 
>> 
>> I did some profiling today, here are some specific findings:
>> 1, CPUs used for GC is around 35% after we enabled soft memory limit, and it 
>> was 45%+ before. I don't have too much experience here on how much CPU we 
>> should spend on GCs ideally, but my intuition 35% is pretty high.
>> 2, For GC, most of the CPU is on runtime.scanObject which I guess is 
>> dependent on how many object we allocate and how fast that is. 
>> 3, With some further look at the heap profiling, it turns out most of the 
>> objects (70%+) allocated are due to complex protobuf messages we use for 
>> communications between services which can be big and might have deep-nested 
>> submessages.
>> 
>> On Monday, October 30, 2023 at 2:19:23 PM UTC-7 Michael Knyszek wrote:
>> I second Jason's message, and +1 to off-heap memory as a last resort.
>> Yes, indeed. One of the advantage using Go is we don't need to manage memory 
>> by ourselves, I will try other options first and see how much we can 
>> improve. 
>> Here are a few more details:
>> 
>> For a starting point on how to reduce memory allocations directly, see 
>> https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require 
>> restructuring your program in places. (e.g. passing byte slices to functions 
>> to be filled instead of returning byte slices; that sort of thing.)
>> RE: pooling memory, take a look look at sync.Pool 
>> (https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at 
>> reducing the number of memory allocations that are made in the steady-state.
>> Object pooling is actually one of the most promising option we are trying to 
>> implement right now. One quick question: is sync.Pool also feasible for 
>> complex protobuf messages? any pitfall we should be take into consideration? 
>>> 
>>> On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote:
>>> Similar to Robert's suggestion, you could just use non-GC-ed memory within 
>>> the process.
>>> 
>>> https://github.com/glycerine/offheap provides an example. 
>>> 
>>> The central idea is that the Go GC will never touch memory that you have 
>>> requested
>>> yourself from the OS. So you can make your own Arenas. 
>>> https://en.wikipedia.org/wiki/Region-based_memory_management
>>> 
>>> But I would save these as last resorts of course. Before that:
>>> 
>>> a) can you reduce the objects allocated per request?  
>>> b) can you allocate everything else on the stack? There are flags to see 
>>> why things are escaping to the heap, use those in your analysis.
>>> (This is by far the simplest and fastest thing. Since the stack is 
>>> automatically unwound when the user request finishes, typically, there is 
>>> no GC to do.)
>>> Will try this out and let you know if we have interesting findings here. 
>>> c) can you allocate a pool of objects that is just reused instead of 
>>> allocating for each new user request?
>>&g

Re: [go-nuts] Suggestions on optimizing Go GC

[Robert Engels]

What is the average wall time of a request?

Based on what you wrote it appears that handling a single request generates a 
lot of garbage - high allocation rate - and for this to be significant I 
suspect the runtime is also significant - which implies to me a spawn and 
destroy request handler is your best bet. 

> On Oct 30, 2023, at 11:56 PM, Zhihui Jiang  wrote:
> 
> Hi Michael, Jason and Robert, thanks a lot for the replies and suggestions!
> 
> I did some profiling today, here are some specific findings:
> 1, CPUs used for GC is around 35% after we enabled soft memory limit, and it 
> was 45%+ before. I don't have too much experience here on how much CPU we 
> should spend on GCs ideally, but my intuition 35% is pretty high.
> 2, For GC, most of the CPU is on runtime.scanObject which I guess is 
> dependent on how many object we allocate and how fast that is. 
> 3, With some further look at the heap profiling, it turns out most of the 
> objects (70%+) allocated are due to complex protobuf messages we use for 
> communications between services which can be big and might have deep-nested 
> submessages.
> 
> On Monday, October 30, 2023 at 2:19:23 PM UTC-7 Michael Knyszek wrote:
> I second Jason's message, and +1 to off-heap memory as a last resort.
> Yes, indeed. One of the advantage using Go is we don't need to manage memory 
> by ourselves, I will try other options first and see how much we can improve. 
> Here are a few more details:
> 
> For a starting point on how to reduce memory allocations directly, see 
> https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require 
> restructuring your program in places. (e.g. passing byte slices to functions 
> to be filled instead of returning byte slices; that sort of thing.)
> RE: pooling memory, take a look look at sync.Pool 
> (https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at 
> reducing the number of memory allocations that are made in the steady-state.
> Object pooling is actually one of the most promising option we are trying to 
> implement right now. One quick question: is sync.Pool also feasible for 
> complex protobuf messages? any pitfall we should be take into consideration? 
> 
> On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote:
> Similar to Robert's suggestion, you could just use non-GC-ed memory within 
> the process.
> 
> https://github.com/glycerine/offheap provides an example. 
> 
> The central idea is that the Go GC will never touch memory that you have 
> requested
> yourself from the OS. So you can make your own Arenas. 
> https://en.wikipedia.org/wiki/Region-based_memory_management
> 
> But I would save these as last resorts of course. Before that:
> 
> a) can you reduce the objects allocated per request?  
> b) can you allocate everything else on the stack? There are flags to see why 
> things are escaping to the heap, use those in your analysis.
> (This is by far the simplest and fastest thing. Since the stack is 
> automatically unwound when the user request finishes, typically, there is no 
> GC to do.)
> Will try this out and let you know if we have interesting findings here. 
> c) can you allocate a pool of objects that is just reused instead of 
> allocating for each new user request?
> d) Is there anything that can be effectively cached and re-used instead of 
> allocated?
> Good point! We actually have an in-memory cache which already haas very high 
> cache hit ratio of 95%+. Seems not too much headroom here to further reduce 
> the CPUs on GC. 
> 
> Use the profiler pprof to figure out what is going on.
> Thanks! pprof indeed is very helpful tool and the problem we are facing seems 
> to boil down to too many large/complex protobuf message passed around 
> different services which allocates too many objects during the proto 
> unmarshal. 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/2e3ac44e-923b-4b6b-88ec-743f8474c83an%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0A66E87C-1BD6-4486-B152-763188B404C7%40ix.netcom.com.

Re: [go-nuts] Suggestions on optimizing Go GC

[Robert Engels]

If the objects are created and discarded you might be able to spawn a new 
process to handle the request - or maybe a bunch - then just kill it and start 
a new process. Possibly use shared memory for constant data. 

This is the poor man’s generational garbage collector. 

> On Oct 29, 2023, at 9:43 PM, Zhihui Jiang  wrote:
> 
> Hi there,
> 
> We have a large-scale recommendation system serving millions of users which 
> is built using Golang. It has worked well until recently when we are trying 
> to enlarge our index or candidate pool by 10X in which case the number of 
> candidate objects created to serve each user request can also increase by 
> 5~10X. Those huge number of objects created on heap cause a big jump of the 
> CPU used for GC itself and thus significantly reduces the system throughput.
> 
> We have tried different ways to reduce GC cost, like using soft memory limit 
> and dynamically tuning the value of GOGC similar to what is described here. 
> Those indeed helped, but they won't reduce the intrinsic cost of GC because 
> the huge number of objects in heap have to be recycled anyway. 
> 
> I'm wondering if you have any suggestions about how to reduce object 
> allocations during request serving?
> 
> Thanks!
> Best
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/a6b8f58f-9452-43e6-9e63-92d944dd0caan%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5741D093-86FC-41F0-BBF1-E4106C92ABC6%40ix.netcom.com.

Re: [go-nuts] Re: Is it expected that signal.NotifyContext() changes the execution thread?

[Robert Engels]

Almost all graphics systems are single threaded. Some allow additional non 
rendering event loops but with heavy restrictions that GUI components are only 
accessed on the main event loop thread. 

> On Oct 3, 2023, at 2:00 AM, 'wagner riffel' via golang-nuts 
>  wrote:
> 
> On Tue Oct 3, 2023 at 05:54 AM UTC, Kurtis Rader wrote:
> > Thank you to Ian and TheDiveO. I don't understand why functions like
> > gocv.io/x/gocv.NewWindow() have to run on the initial OS thread (at least
> > on macOS).
> 
> It's common for C and C++ libraries to use TLS (thread local storage)
> to attach data/state to each thread, one example is the global errno
> variable used in C to signal errors, each thread read/write its own
> variable even though in code they are wirting "the same" variable,
> using libc from cgo and reading errno for failures would give wrong
> results if the goroutine moved OS threads. It's unrelated which thread
> it is and that's why it's not a default, you could start NewWindow at
> some point that its goroutine is running in thread4, and if it's not
> pinned to run in thread4 you have the same issue with the "initial
> thread".
> 
> ps: Specific with graphics, I know OpenGL retains thread-local data,
> which might explain why libraries that have this common ancestor needs
> to LockOSThread, I'm not sure about Mac and Windows.
> 
> -w
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/dcbc6ba0-54c8-5942-eb4d-120ba8d65f03%40104d.net.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/DD240547-61CF-40AC-9DDC-C49504203FB4%40ix.netcom.com.

Re: [go-nuts] Re: Help with WASM performance

[Robert Engels]

Typically WASM performance is compared to JavaScript - ie. faster than 
JavaScript usually 1.5-10x. 

People don’t usually compare WASM with native apps. 

> On Sep 22, 2023, at 7:04 PM, Robert Engels  wrote:
> 
> 
> WASM goes through the browser - so it is very different. Are you using OpenGL 
> or similar?
> 
>>> On Sep 22, 2023, at 3:44 PM, Stephen Illingworth 
>>>  wrote:
>>> 
>> 
>> I'm comparing the results of a program compiled for AMD64 and WASM 
>> architectures. The code is the same except for the change of architecture. 
>> The size of the difference in performance is unexpected to me but maybe it's 
>> normal.
>> 
>>> On Friday, 22 September 2023 at 20:16:20 UTC+1 Robert Engels wrote:
>>> When you say negative performance are you talking about a native app 
>>> running directly on the hardware what are you referring to exactly
>>> 
>>>>> On Sep 22, 2023, at 1:03 PM, Stephen Illingworth  
>>>>> wrote:
>>>>> 
>>>> 
>>> 
>>>> I've been thinking some more about this problem this week. I found the 
>>>> performance profiler in Chrome and can see that the each frame is taking 
>>>> longer than the required 16.7ms to create. The duration for each frame is 
>>>> more like 100ms. The native performance meanwhile can reach about 7ms. I 
>>>> expected a drop in performance but not that much.
>>>> 
>>>> If anyone can offer any insight or if these figures seem wrong then I'd 
>>>> love to take some advice.
>>>> 
>>>> Regards
>>>> Stephen
>>>> 
>>>>>> On Saturday, 16 September 2023 at 09:02:34 UTC+1 Stephen Illingworth 
>>>>>> wrote:
>>>>>> I have tried running it in Firefox and Chromium. The webserve.sh script 
>>>>>> in the ebiten_test folder runs a small httpd server to serve up the 
>>>>>> binary to the browser.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Saturday, 16 September 2023 at 08:45:52 UTC+1 Brian Candler wrote:
>>>>>>> What WASM runtime are you using to execute the code?
>>>>> 
>>>> -- 
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to golang-nuts...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/golang-nuts/d8926ffb-9ca3-48ac-aabc-76a8a43947dan%40googlegroups.com.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/1123a2da-3862-4da8-baf5-bc67dd2b7e6en%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3C5CC55B-8356-4DF5-8E56-9CC067618A38%40ix.netcom.com.

Re: [go-nuts] Re: Help with WASM performance

[Robert Engels]

WASM goes through the browser - so it is very different. Are you using OpenGL 
or similar?

> On Sep 22, 2023, at 3:44 PM, Stephen Illingworth 
>  wrote:
> 
> 
> I'm comparing the results of a program compiled for AMD64 and WASM 
> architectures. The code is the same except for the change of architecture. 
> The size of the difference in performance is unexpected to me but maybe it's 
> normal.
> 
>> On Friday, 22 September 2023 at 20:16:20 UTC+1 Robert Engels wrote:
>> When you say negative performance are you talking about a native app running 
>> directly on the hardware what are you referring to exactly
>> 
>>>> On Sep 22, 2023, at 1:03 PM, Stephen Illingworth  
>>>> wrote:
>>>> 
>>> 
>> 
>>> I've been thinking some more about this problem this week. I found the 
>>> performance profiler in Chrome and can see that the each frame is taking 
>>> longer than the required 16.7ms to create. The duration for each frame is 
>>> more like 100ms. The native performance meanwhile can reach about 7ms. I 
>>> expected a drop in performance but not that much.
>>> 
>>> If anyone can offer any insight or if these figures seem wrong then I'd 
>>> love to take some advice.
>>> 
>>> Regards
>>> Stephen
>>> 
>>>>> On Saturday, 16 September 2023 at 09:02:34 UTC+1 Stephen Illingworth 
>>>>> wrote:
>>>>> I have tried running it in Firefox and Chromium. The webserve.sh script 
>>>>> in the ebiten_test folder runs a small httpd server to serve up the 
>>>>> binary to the browser.
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Saturday, 16 September 2023 at 08:45:52 UTC+1 Brian Candler wrote:
>>>>>> What WASM runtime are you using to execute the code?
>>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/d8926ffb-9ca3-48ac-aabc-76a8a43947dan%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1123a2da-3862-4da8-baf5-bc67dd2b7e6en%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4B64C9B0-44CB-49A6-8A0C-4914595DB41A%40ix.netcom.com.

Re: [go-nuts] Re: Help with WASM performance

[Robert Engels]

When you say negative performance are you talking about a native app running 
directly on the hardware what are you referring to exactly

> On Sep 22, 2023, at 1:03 PM, Stephen Illingworth 
>  wrote:
> 
> 
> I've been thinking some more about this problem this week. I found the 
> performance profiler in Chrome and can see that the each frame is taking 
> longer than the required 16.7ms to create. The duration for each frame is 
> more like 100ms. The native performance meanwhile can reach about 7ms. I 
> expected a drop in performance but not that much.
> 
> If anyone can offer any insight or if these figures seem wrong then I'd love 
> to take some advice.
> 
> Regards
> Stephen
> 
>> On Saturday, 16 September 2023 at 09:02:34 UTC+1 Stephen Illingworth wrote:
>> I have tried running it in Firefox and Chromium. The webserve.sh script in 
>> the ebiten_test folder runs a small httpd server to serve up the binary to 
>> the browser.
>> 
>> 
>> 
>>> On Saturday, 16 September 2023 at 08:45:52 UTC+1 Brian Candler wrote:
>>> What WASM runtime are you using to execute the code?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/d8926ffb-9ca3-48ac-aabc-76a8a43947dan%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/BEAAB244-A30A-46D0-B6AB-B155091E12BA%40ix.netcom.com.

Re: [go-nuts] x/mobile: avoid to catch sigabort signal

[Robert Engels]

This most likely means your other code is corrupting the go runtime/heap/stack. 

I would run other tests on this code exclusively (asan, etc). 

> On Sep 10, 2023, at 12:54 PM, Danilo bestbug  
> wrote:
> 
> 
> Hey TheDiveO, thanks for answering. 
> My focus is not to hide the signal but I would like to understand if this 
> behavior is wanted and in that case what we can do to avoid to have crash 
> report where the crashed thread does not run any of our "code". 
> 
> Il giorno domenica 10 settembre 2023 alle 18:12:37 UTC+2 TheDiveO ha scritto:
>> Maybe this SO Q with A might also help with further details: 
>> https://stackoverflow.com/questions/47869988/how-does-cgo-handle-signals
>> 
>>> On Friday, September 8, 2023 at 11:38:38 PM UTC+2 Danilo bestbug wrote:
>>> Ehy Ian, thanks for the response. Apology if I was not clear, I try to 
>>> explain in a different way but it's hard for me too since I'm not 100% 
>>> confident about what it's happening here exactly, I'm tring to follow the 
>>> trace but any feedback is more than welcome.
>>> The problem I'm facing is the following: I've a small utility written in GO 
>>> and integrated in an app iOS as SDK. At the moment if I've undestood 
>>> correctly from the thread I've linked the GO runtime are cacthing a 
>>> sigabort signal that are not from the GO stack but it's generated from 
>>> another thread with the result that it's look like the GO runtime is 
>>> crashing from the apple report.
>>> 
>>> If this behavior of the GO runtime is legittime when GO is an executable in 
>>> my context is a problem since the developer follow the GO stack instead of 
>>> the other thread stack.
>>> 
>>> Now what I'm try to understand if this behavior can be somehow change, and 
>>> if so how should I do?
>>> Il giorno venerdì 8 settembre 2023 alle 22:34:07 UTC+2 Ian Lance Taylor ha 
>>> scritto:
 On Thu, Sep 7, 2023 at 11:41 PM Danilo bestbug 
  wrote: 
 > 
 > Some weeks ago I've opened a possible bug on github and the only 
 > response I received is a reference to 
 > "This looks like the program (the Go runtime, or not) intentionally 
 > crashing when it is already in a bad condition, like receiving an 
 > unhandled signal on a non-Go thread." 
 > 
 > I would like to stop the GO system to do this kind of behaviour 
 > (intercepting unhandled signal) otherwise the team who work on the crash 
 > keep searching the problem on the GO thread crashed instead of 
 > elsewhere. This for us is a big problem and I love if someone can help 
 > me to address this matter! 
 
 I'm sorry, I don't really understand what you are asking. What I can 
 tell you is that signal handling in Go programs is managed via the 
 os/signal package. See https://pkg.go.dev/os/signal. 
 
 Ian 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/88739304-c34e-4a56-b7b4-85d763f71947n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/01BB404E-7755-4F3B-9041-A48EC3D86C1E%40ix.netcom.com.

Re: [go-nuts] Re: Best IDE for GO ?

[Robert Engels]

Sorry that was snotty. Bring back the punch cards to really prove who’s a great 
developer!

> On Aug 26, 2023, at 4:29 PM, Robert Engels  wrote:
> 
> 
> Or you click on the top stack line in an IDE and then arose down. 
> 
> It’s like some people want to go back to horse and buggy “because it was 
> better for the environment”. Balancing competing priorities is a great skill 
> to have. 
> 
>>> On Aug 26, 2023, at 3:51 PM, Justin Israel  wrote:
>>> 
>> 
>> 
>> 
>>> On Sun, Aug 27, 2023 at 12:47 AM Mike Schinkel  
>>> wrote:
>>> If I understand what you are asking then JetBrains GoLand does. 
>>> 
>>> I do not know if there is a way to use the keyboard, but it does provides 
>>> links you can click when it displays the call stack on panic.
>> 
>> If your keymap configuration derives from one of the system configs 
>> (Windows, Macos, ...) then "Next Frame" and "Previous Frame" have default 
>> mapped hotkeys. So you can do the same workflow Jason mentioned, starting a 
>> debugger execution, which panics, and then navigating the stack all via 
>> keyboard (if that really is your thing).
>>  
>>> 
>>> -Mike
>>> 
>>>> On Saturday, August 26, 2023 at 8:33:08 AM UTC-4 Jason E. Aten wrote:
>>>> Is there any IDE that allows you to jump through a stack trace like emacs 
>>>> does?
>>>> 
>>>> e.g. If I have a panic on a run, with two keystrokes I can jump to the 
>>>> origin
>>>> of the panic, and then their caller, and then the parent caller, and then 
>>>> up to
>>>> the grandparent on the call stack... instantly. I've never found this 
>>>> essential
>>>> functionality elsewhere, but maybe I'm just not familiar... A friend of 
>>>> mine tried
>>>> to add it to Visual Studio and gave up... it was just too hard for VS. But 
>>>> maybe JetBrains has it??
>>>> 
>>>> I'd love to try an IDE other than emacs, but this is a fundament thing, 
>>>> that I cannot give up.
>>>> 
>>>>> On Friday, August 25, 2023 at 6:21:35 PM UTC+1 Mike Schinkel wrote:
>>>>> Yes, as Luke Crook mentioned I think those requirements are more ALM 
>>>>> functionality than IDE functionality.  
>>>>> 
>>>>> Generally, ALM addresses concerns broader than individual concerns 
>>>>> whereas IDEs are more focused on individual productivity.
>>>>> 
>>>>> Just my opinion, but I would expect you'd be better off finding an ALM 
>>>>> solution and then an IDE that integrates with that ALM, or vice versa, 
>>>>> i.e. find an IDE that integrates with an ALM and then use that ALM.
>>>>> 
>>>>> #fwiw
>>>>> 
>>>>> -Mike
>>>>>> On Wednesday, August 23, 2023 at 7:21:46 AM UTC-4 alex-coder wrote:
>>>>>> Hi All !
>>>>>> 
>>>>>> Considering that IBM's punch cards were bearing at least twice, I would 
>>>>>> vote for them. :-)
>>>>>> 
>>>>>> Of cource I do agree with them who wrote that to feel comfortable "under 
>>>>>> fingers" is great !
>>>>>> 
>>>>>> So, the tasks to code - they are different. 
>>>>>> Sometimes it is possible to keep all the details regards to the task in 
>>>>>> a head or several.
>>>>>> Sometimes it is nesessary to write say a hard copy of them(details) on a 
>>>>>> paper with a different size.
>>>>>> 
>>>>>> But in case the task from the area of the "poorly formalized". You spend 
>>>>>> paper quickly. :-)
>>>>>> 
>>>>>> The Luke Crook points to:
>>>>>> https://en.wikipedia.org/wiki/Application_lifecycle_management
>>>>>> 
>>>>>> I will simplify the task somewhat and take from ALM for example even 
>>>>>> less than SDLC, namely:
>>>>>> requirements, design, implementation, testing, deployment.
>>>>>> 
>>>>>> 1. Requirements must be described somewhere.
>>>>>> 2. Design artifacts should reflect requirements.
>>>>>> 3. Design decisions refer to objects and messages that
>>>>>> implemented in the form of classes and operations.
>>>>>> 4. Each operation must pass at lea

Re: [go-nuts] Re: Best IDE for GO ?

[Robert Engels]

Or you click on the top stack line in an IDE and then arose down. 

It’s like some people want to go back to horse and buggy “because it was better 
for the environment”. Balancing competing priorities is a great skill to have. 

> On Aug 26, 2023, at 3:51 PM, Justin Israel  wrote:
> 
> 
> 
> 
>> On Sun, Aug 27, 2023 at 12:47 AM Mike Schinkel  
>> wrote:
>> If I understand what you are asking then JetBrains GoLand does. 
>> 
>> I do not know if there is a way to use the keyboard, but it does provides 
>> links you can click when it displays the call stack on panic.
> 
> If your keymap configuration derives from one of the system configs (Windows, 
> Macos, ...) then "Next Frame" and "Previous Frame" have default mapped 
> hotkeys. So you can do the same workflow Jason mentioned, starting a debugger 
> execution, which panics, and then navigating the stack all via keyboard (if 
> that really is your thing).
>  
>> 
>> -Mike
>> 
>>> On Saturday, August 26, 2023 at 8:33:08 AM UTC-4 Jason E. Aten wrote:
>>> Is there any IDE that allows you to jump through a stack trace like emacs 
>>> does?
>>> 
>>> e.g. If I have a panic on a run, with two keystrokes I can jump to the 
>>> origin
>>> of the panic, and then their caller, and then the parent caller, and then 
>>> up to
>>> the grandparent on the call stack... instantly. I've never found this 
>>> essential
>>> functionality elsewhere, but maybe I'm just not familiar... A friend of 
>>> mine tried
>>> to add it to Visual Studio and gave up... it was just too hard for VS. But 
>>> maybe JetBrains has it??
>>> 
>>> I'd love to try an IDE other than emacs, but this is a fundament thing, 
>>> that I cannot give up.
>>> 
 On Friday, August 25, 2023 at 6:21:35 PM UTC+1 Mike Schinkel wrote:
 Yes, as Luke Crook mentioned I think those requirements are more ALM 
 functionality than IDE functionality.  
 
 Generally, ALM addresses concerns broader than individual concerns whereas 
 IDEs are more focused on individual productivity.
 
 Just my opinion, but I would expect you'd be better off finding an ALM 
 solution and then an IDE that integrates with that ALM, or vice versa, 
 i.e. find an IDE that integrates with an ALM and then use that ALM.
 
 #fwiw
 
 -Mike
> On Wednesday, August 23, 2023 at 7:21:46 AM UTC-4 alex-coder wrote:
> Hi All !
> 
> Considering that IBM's punch cards were bearing at least twice, I would 
> vote for them. :-)
> 
> Of cource I do agree with them who wrote that to feel comfortable "under 
> fingers" is great !
> 
> So, the tasks to code - they are different. 
> Sometimes it is possible to keep all the details regards to the task in a 
> head or several.
> Sometimes it is nesessary to write say a hard copy of them(details) on a 
> paper with a different size.
> 
> But in case the task from the area of the "poorly formalized". You spend 
> paper quickly. :-)
> 
> The Luke Crook points to:
> https://en.wikipedia.org/wiki/Application_lifecycle_management
> 
> I will simplify the task somewhat and take from ALM for example even less 
> than SDLC, namely:
> requirements, design, implementation, testing, deployment.
> 
> 1. Requirements must be described somewhere.
> 2. Design artifacts should reflect requirements.
> 3. Design decisions refer to objects and messages that
> implemented in the form of classes and operations.
> 4. Each operation must pass at least one test.
> All tests must be passed successfully.
> 5. The application is assembled and installed there and 
> the tests are successfully passed again.
> 
> Question: is there any IDE or plugin which one support that kind of 
> dependencies in a graphical mode ?
> 
> Thank you.
> 
> вторник, 22 августа 2023 г. в 18:22:52 UTC+3, Mike Schinkel: 
>> On Saturday, August 19, 2023 at 5:27:34 AM UTC-4 alex-coder wrote:
>> What I'm looking for is the ability to manage dependencies not only in 
>> code,
>> but entirely in a project from requirements to deployment.
>> 
>> I assume you mean a lot more than just Go package dependencies, as `go 
>> mod` handles those nicely.
>> 
>> Can you elaborate on the specific dependencies you are trying to manage? 
>>  In specific, vs generalities.
>> 
>> -Mike  
>> 
>> -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/golang-nuts/4yVXlyZZatM/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/c8b2e710-240b-4f4a-ae80-3142578673d3n%40googlegroups.com.
> 
> -- 
> You received this 

Re: [go-nuts] Best IDE for GO ?

[Robert Engels]

Ah. I forgot I was on that Internet thingy. 

> On Aug 23, 2023, at 2:29 AM, TheDiveO  wrote:
> 
> "That education place" considers thermodynamics to be a cultural issue, see 
> their waterseer disaster where MIT even doubled down on not understanding 
> physics and thermodynamics at all.
> 
>> On Sunday, August 20, 2023 at 1:57:22 PM UTC+2 Robert Engels wrote:
>> MIT = “that education place” is pretty funny. But you’re spot on. 
>> 
>>>> On Aug 20, 2023, at 2:51 AM, TheDiveO  wrote:
>>>> 
>>> That education place has never talked to any employer, that's what their 
>>> list suggests. It's not about the items on this list. It won't ever 
>>> correct. But it is basically saying that with one or a few more classes 
>>> you're ready for your job. That's marketing selling. Depending on the job 
>>> there's need for cheap code producers, or in other jobs for people going 
>>> where there is no prior art. Totally different coders. But both don't need 
>>> coders with many classes, either because it's making them expensive or they 
>>> have gone through the classes to just accumulate useless knowledge they 
>>> still cannot transfer to new challenges.
>> 
>>> 
>>> 
>>>>> On Saturday, August 19, 2023 at 8:54:38 PM UTC+2 Andrew Harris wrote:
>>>>> The kinds of skills and knowledge covered by 
>>>>> https://missing.csail.mit.edu are important and hard to gain from an IDE. 
>>>>> I think that's the badge of competence earned here.
>>>>> 
>>>>> On Saturday, August 19, 2023 at 11:21:44 AM UTC-7 Robert Engels wrote:
>>>>> The power of IDEs is the ease of refactoring and integration with other 
>>>>> build tools (git, lint, github/gerrit). If you’d added all of these 
>>>>> plugins to vim - you’ve created your own ide - and it probably pales in 
>>>>> comparison to a real IDE.
>>>>> 
>>>>> Using plain vim as a badge of competence was disproven long ago.
>>>> 
>>> -- 
>> 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/78b49e14-d8e4-43eb-a4e0-803ed48d3669n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/6afd29fb-4215-4942-b104-a96e7a181ca7n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5627B5D0-41A2-4DD4-878F-94A44AA28A61%40ix.netcom.com.

Re: [go-nuts] Best IDE for GO ?

[Robert Engels]

Depends on the size of the org. If you encourage pair programming on even 
“active code review” - not having a standardized ide it a problem. Also larger 
org add in their own tooling integrations - and doing that for multiple IDEs is 
not cost effective. 

> On Aug 20, 2023, at 2:52 AM, TheDiveO  wrote:
> 
> well, our "(major) engineering orgs" leave the choice of IDE to our devs. 
> Devs have different styles, so as long as they meed the demand, who cares.
> 
>> On Saturday, August 19, 2023 at 11:17:35 PM UTC+2 Robert Engels wrote:
>> Reread what I wrote. Vim with autocomplete, etc is not a simple text editor 
>> with syntax coloring. 
>> 
>> Still every major software engineering org in the world uses an ide (or 
>> multiple). I guess they don’t know what they are doing. 
>> 
>> Btw, Googles current IDE is based on VSCode. 
>> 
>> > On Aug 19, 2023, at 3:24 PM, Jan Mercl <0xj...@gmail.com> wrote: 
>> > 
>> > On Sat, Aug 19, 2023 at 10:06 PM Christian Stewart 
>> >  wrote: 
>> > 
>> >> Autocomplete and a go language server (gopls) add a ton of speed because 
>> >> you don't need to look up the docs for function and variable names. And 
>> >> go to definition improves speed navigating code significantly. 
>> > 
>> > - Using autocomplete and go-to-definiton does not require VSCode or 
>> > any other IDE. 
>> > - I do use autocomplete and go-to-definition. When I said I use no 
>> > IDE, that does not mean I don't use those features. 
>> > - The speed of typing/inputting code is overally a rather negligible 
>> > factor of the total time cost of developing/debugging and maintaining 
>> > any non-toy project. IOW, it's not a significant metric of 
>> > productivity. 
>> > 
>> >> But vim-go can do both, so why not just use it? 
>> > 
>> > Because I use govim? ;-) 
>> > 
>> > (But not for my large projects, gopls chokes on them still too often 
>> > to tolerate it.) 
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google Groups 
>> > "golang-nuts" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send an 
>> > email to golang-nuts...@googlegroups.com. 
>> > To view this discussion on the web visit 
>> > https://groups.google.com/d/msgid/golang-nuts/CAA40n-XusymW6gb5OnDa_7QWAWPFSkwKYQMYUm-d7419EZ%2BGkQ%40mail.gmail.com.
>> >  
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/8d5b52de-ddee-461a-9ed5-9b0968382d17n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A3D0B8C1-14D6-4E8F-8074-E244D6CB42C9%40ix.netcom.com.

Re: [go-nuts] Best IDE for GO ?

[Robert Engels]

MIT = “that education place” is pretty funny. But you’re spot on. 

> On Aug 20, 2023, at 2:51 AM, TheDiveO  wrote:
> 
> That education place has never talked to any employer, that's what their 
> list suggests. It's not about the items on this list. It won't ever correct. 
> But it is basically saying that with one or a few more classes you're ready 
> for your job. That's marketing selling. Depending on the job there's need for 
> cheap code producers, or in other jobs for people going where there is no 
> prior art. Totally different coders. But both don't need coders with many 
> classes, either because it's making them expensive or they have gone through 
> the classes to just accumulate useless knowledge they still cannot transfer 
> to new challenges.
> 
>> On Saturday, August 19, 2023 at 8:54:38 PM UTC+2 Andrew Harris wrote:
>> The kinds of skills and knowledge covered by https://missing.csail.mit.edu 
>> are important and hard to gain from an IDE. I think that's the badge of 
>> competence earned here.
>> 
>> On Saturday, August 19, 2023 at 11:21:44 AM UTC-7 Robert Engels wrote:
>> The power of IDEs is the ease of refactoring and integration with other 
>> build tools (git, lint, github/gerrit). If you’d added all of these plugins 
>> to vim - you’ve created your own ide - and it probably pales in comparison 
>> to a real IDE.
>> 
>> Using plain vim as a badge of competence was disproven long ago.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/78b49e14-d8e4-43eb-a4e0-803ed48d3669n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/86397B14-BEF6-420F-885B-3C0FEEBE5EA8%40ix.netcom.com.

Re: [go-nuts] Best IDE for GO ?

[Robert Engels]

Reread what I wrote. Vim with autocomplete, etc is not a simple text editor 
with syntax coloring. 

Still every major software engineering org in the world uses an ide (or 
multiple). I guess they don’t know what they are doing. 

Btw, Googles current IDE is based on VSCode. 

> On Aug 19, 2023, at 3:24 PM, Jan Mercl <0xj...@gmail.com> wrote:
> 
> On Sat, Aug 19, 2023 at 10:06 PM Christian Stewart
>  wrote:
> 
>> Autocomplete and a go language server (gopls) add a ton of speed because you 
>> don't need to look up the docs for function and variable names. And go to 
>> definition improves speed navigating code significantly.
> 
> - Using autocomplete and go-to-definiton does not require VSCode or
> any other IDE.
> - I do use autocomplete and go-to-definition. When I said I use no
> IDE, that does not mean I don't use those features.
> - The speed of typing/inputting code is overally a rather negligible
> factor of the total time cost of developing/debugging and maintaining
> any non-toy project. IOW, it's not a significant metric of
> productivity.
> 
>> But vim-go can do both, so why not just use it?
> 
> Because I use govim? ;-)
> 
> (But not for my large projects, gopls chokes on them still too often
> to tolerate it.)
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAA40n-XusymW6gb5OnDa_7QWAWPFSkwKYQMYUm-d7419EZ%2BGkQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/E97615C4-8848-4749-96AF-C2AD95EA20E5%40ix.netcom.com.

Re: [go-nuts] Best IDE for GO ?

[Robert Engels]

I guarantee that two developers of equal competence - the one with a powerful 
IDE will outperform the other using a text editor with syntax highlighting (but 
come on that is a crutch a real developer doesn’t need) by 2-10x depending on 
the task. 

You simply cannot work on large complex systems without the assistance of an 
IDE. 

If anyone thinks their ability to code without using an ide makes them a better 
developer they are sadly mistaken. 

If you’re coding a 20 line leet code solution you can skip the ide - but that 
isn’t modern software engineering in any aspect. 

> On Aug 19, 2023, at 1:54 PM, Andrew Harris  wrote:
> 
> The kinds of skills and knowledge covered by https://missing.csail.mit.edu 
> are important and hard to gain from an IDE. I think that's the badge of 
> competence earned here.
> 
> On Saturday, August 19, 2023 at 11:21:44 AM UTC-7 Robert Engels wrote:
> The power of IDEs is the ease of refactoring and integration with other build 
> tools (git, lint, github/gerrit). If you’d added all of these plugins to vim 
> - you’ve created your own ide - and it probably pales in comparison to a real 
> IDE.
> 
> Using plain vim as a badge of competence was disproven long ago.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1d5b25ca-7769-4d28-96ad-443a09ab4e18n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4943BF61-4043-4D56-8693-F6631146AB73%40ix.netcom.com.

Re: [go-nuts] Best IDE for GO ?

[Robert Engels]

The power of IDEs is the ease of refactoring and integration with other build tools (git, lint, github/gerrit). If you’d added all of these plugins to vim - you’ve created your own ide - and it probably pales in comparison to a real IDE.Using plain vim as a badge of competence was disproven long ago. On Aug 19, 2023, at 1:10 PM, William Rehwinkel  wrote:
  

  
  
I use vim for go with the standard syntax
  highlighting and gofmt. With gofmt catching errors when I save,
  compiler errors and looking up things with godoc, I don't see the
  need for anything else. But that's just me

On 8/19/23 05:27, alex-coder wrote:


  
  Hi All !
  Gophers, there is at least 10 years as GO on a market, Good job !
  
  I found a list of the best IDE and Plugins there:
  https://github.com/golang/go/wiki/IDEsAndTextEditorPlugins
  
  As I remember Java came around 1995 and within 3-4 years several
  companies 
  developed really great IDEs successful or less to develop
  commercial project 
  on Java. Here is a list of but I may miss somewhat, sorry, 
  it was relatively a long time ago:
  
  Forte for Java ( Praga, Czech republic)
  VisualCafe (Symantec)
  VisuailAge (IBM)
  JBuilder (Borland)
  Together Control Center (TogetherSoft, Germany)
  Eclipse, and set of commercial IDE on a base on: RSA,RSD,WID and
  so on (IBM)
  
  appear later:
  
  VS Code ( Microsoft)
  IDEA (JetBrain)
  
  What I'm looking for is the ability to manage dependencies not
  only in code, 
  but entirely in a project from requirements to deployment.
  The feature should be inbuild into IDE or at least it would be
  possible 
  to write plugin to implement it.
  
  It is quite possible that I'm missing somewhat here.
  Gophers, may be there is another place where I should look for IDE
  for GO ?
  
  Thank you.
  -- 
  You received this message because you are subscribed to the Google
  Groups "golang-nuts" group.
  To unsubscribe from this group and stop receiving emails from it,
  send an email to golang-nuts+unsubscr...@googlegroups.com.
  To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/61fefbf1-bb6d-4d90-b77b-a94c70582826n%40googlegroups.com.


-- 
+ --- +
|   William Rehwinkel - Oberlin College and   |
|  Conservatory '24   |
|will...@williamrehwinkel.net |
| PGP key:|
| https://ftp.williamrehwinkel.net/pubkey.txt |
+ --- +

  




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2533626d-cfb0-9a38-08a7-f5f8918aa2a0%40gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/504F48FA-7F27-4B76-B96A-DDB8441C2F8F%40ix.netcom.com.


OpenPGP_signature
Description: Binary data




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/504F48FA-7F27-4B76-B96A-DDB8441C2F8F%40ix.netcom.com.

Re: [go-nuts] Best IDE for GO ?

[Robert Engels]

I would just have chatgpt write all your Go code. 

> On Aug 19, 2023, at 7:41 AM, peterGo  wrote:
> 
> 
> On Saturday, August 19, 2023 at 6:23:39 AM UTC-4 TheDiveO wrote:
> Nothing better than an IBM card puncher as my IDE!  
> 
>  I prefer a Teletype Model 33 ASR.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/2cc6ff26-e1cf-481b-972f-9f110ec78a0dn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/00258F95-BC0D-4497-940D-B0E9F7F97F73%40ix.netcom.com.

Re: [go-nuts] Error handling

[robert engels]

For some perspective. Go’s error handling mimics C (for the most part). They 
had a decade to decide how to improve the error handling when they designed 
C++. They came up with exceptions. Java is C++ like. They had a decade to 
improve error handling. They came up with exceptions + throws.

The Go designers do not want exceptions in any way shape or form, so you’re 
pretty much going to be stuck with what you have (all of the decent proposals 
are “exception like”- exceptions by another name) so learn to love it or use a 
different language.

> On Jul 30, 2023, at 2:02 AM, Brian Candler  wrote:
> 
> Just to be clear: are you hard-coding the variable name "err" into the 
> semantics of "orelse"?  That is, you can't assign the error return to a 
> variable of any other name?
> 
> I disagree that this makes the job of linters any easier than it is today.  
> For example, if you'd written
> 
> ...
> err = io.Copy(w, r)
>   err = w.Close() orelse return err
> }
> 
> then you'd still need to detect "value assigned but not used" in the linter 
> (assuming it doesn't become *compulsory* to use "orelse" on any assignment to 
> a variable called "err")
> 
> On Sunday, 30 July 2023 at 06:57:15 UTC+1 DrGo wrote:
> I looked at the long list of proposals to improve error handling in go but I 
> have not seen the one I am describing below. If I missed a similar , can you 
> pls direct me to where I can find it. If not what do you think of this 
> approach. 
> 
> This involves introducing a new keyword "orelse" that is a syntactic sugar 
> for an "if err!=nil" block.
> 
> The example code in Russ Cox's paper[1] will look something like this:
> 
> func CopyFile(src, dst string) error {
>   r, err := os.Open(src) orelse return err
>   defer r.Close()
>   w, err := os.Create(dst) orelse return err
>   defer w.Close()
> err = io.Copy(w, r) orelse return err
>   err = w.Close() orelse return err
> }
> 
> It is an error to not return an error from an orelse block.
> 
> In my eyes, this has the same explicitness and flexibility of the current 
> style but is significantly less verbose. It permits ignoring the error, 
> returning it as is or wrapping it. Because orelse is not used for any other 
> purpose, it would be easy for reviewers and linters to spot lack of error 
> handling.  
> 
> It also works well with named returns. e.g., 
> 
> func returnsObjorErro() (obj Obj, err error) {
>   obj, err := createObj() orelse return  //returns nil and err
> } 
> 
> otherwise orelse is like "else" so e.g., it can be followed by a block if 
> additional cleanup or error formatting etc is needed before returning, eg 
> w, err := os.Create(dst) orelse {
>   
>   return err 
> }
> 
> Similarity to "else" hopefully means that it is easy to learn. It is 
> obviously backward compatible  
> 
> What do you think?
> 
> [1] 
> https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling-overview.md
>  
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/330e88d9-8072-4614-ae56-8ce9c59517f3n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/60596671-D3AC-49D4-8575-F8EB3D9B6BF6%40ix.netcom.com.

Re: [go-nuts] Please consider voting to reopen Golang subreddit

[Robert Engels]

I’m fairly certain Reddit will license those tools for moderators use. That is 
not what they are trying to prevent. 

> On Jun 24, 2023, at 8:39 AM, Wojciech S. Czarnecki  wrote:
> 
> Dnia 2023-06-23, o godz. 23:50:32
> Amnon  napisał(a):
> 
>> Sorry, but I have been away and missed the context.
> 
>> What is the protest about?
> 
> Reddit's decision to sink all good free tools that allow people (mods) to 
> work several hours per day
> keeping their communities free of propaganda and scams.
> 
>> 
>> What has redit changed?
> 
> APIs that were free to access by third parties will be priced as gold.
> 
>> Thanks
> yw.
> 
> P.S.
> It is worth mentioning that Reddit *has* a good reason to close APIs that are 
> abused by A"I" businesses.
> Just there is no will to make such changes robust, IMO, as it would cost.  It 
> seems to me that current Reddit's
> brass has no faintest idea how many $millions monthly mod's work is worth.
> 
> -- 
> Wojciech S. Czarnecki
> << ^oo^ >> OHIR-RIPE
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/20230624153919.43160eba%40xmint.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C63B3846-FE48-49B7-BEB4-0E55CBADA79D%40ix.netcom.com.

Re: [go-nuts] Locking (`__GI___pthread_mutex_unlock`) takes most of the execution time when using cgo with large number of CPUs

[robert engels]

Since this is an initialization guard - can’t it simply be double checked with 
an atomic flag?

> On Jun 20, 2023, at 9:04 AM, Ian Lance Taylor  wrote:
> 
> On Tue, Jun 20, 2023 at 5:32 AM Bảo Phan Quốc  wrote:
>> 
>> I'm using cgo to call a C function from Go. Inside the C function there is a 
>> callback to a Go function. In other way, I'm calling Go -> C -> Go.
>> 
>> After running pprof, I noticed that the __GI___pthread_mutex_unlock took 
>> half of the execution time. AFAIK, cgo has an overhead, especially calling 
>> back from C to Go. But it's weird that cgo takes half of the execution time 
>> to do some locking. Is there something wrong in my code?
> 
> Every call from C to Go does acquire a mutex to check that the Go
> runtime has been fully initialized.  This is normally not a big deal
> as the mutex is held only briefly.  But your code does a lot of
> parallel calls from C to Go.  It is possible that the heavy contention
> on the mutex is causing the results that you are seeing.
> 
> The code in question is the C function _cgo_wait_runtime_init_done in
> runtime/gco/gcc_libinit.c.  It might be possible to speed it up in
> some way to avoid the mutex in the normal case.
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcU7Csb_hf5gvy8dVHKEHd8GFkN-fUML18uW0HiQGOAXyg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EF9D02CC-01CB-49D6-922E-59766A396927%40ix.netcom.com.

Re: [go-nuts] Re: [RFC] Yet another proposal for Go2 error handling

[Robert Engels]

What I was suggesting was a far narrower scope. I can see how go’s error 
handling makes writing composable math functions more difficult (although I 
would think these would panic not return an error - so then it is trivial). 

If they do need to return errors then using a DSL for the equations and go 
generate you don’t really care how much boilerplate code there is - you don’t 
read or edit it. 

> On Jun 18, 2023, at 3:10 PM, Jan Pfeifer  wrote:
> 
> 
> A DSL + code generator is a longer discussion. But I'm not sure I'm getting 
> the scope of your suggestion right:
> 
> If you refer to having a full ML language DSL and a 
> generator/AOT(ahead-of-time) compiler: folks have tried it before -- see the 
> Swift ML effort by Chris Lattner, now leading Mojo in Modular AI. I don't 
> know the detailed story there, but I thought hard about this (having a 
> separate ML language and generator) before I started the GoMLX project. But 
> from what I can imagine some of the shapes (and hyperparameters) that need 
> checking often are only known at runtime. Also for development and debugging 
> it complicates things: it's super nice to have a full language (Go) in 
> parallel to constructing the graph. 
> 
> If you are referring to a DSL and code generator for error handling only: I'm 
> not sure it would work well because it requires an extra step when developing 
> ... it is just not as convenient / portable -- for instance for IDEs; also 
> I'm often coding in a Jupyter Notebook using GoNB, which would also need 
> modifications. Better if there was some form of support for the language. I'm 
> hoping some error handling proposal would cover this use case.
> 
> 
> On Sun, Jun 18, 2023 at 5:11 PM Robert Engels  wrote:
>> Seems  the easiest way to address that is with a DSL and a code generator. 
>> 
>>> On Jun 18, 2023, at 9:47 AM, Jan Pfeifer  wrote:
>>> 
>>> 
>>> hi Shulhan, I see your points, thanks for the reply. Let me comment on them 
>>> below:
>>> 
>>> On Sat, Jun 17, 2023 at 9:21 AM Shulhan  wrote:
>>>> Hi Jan, thanks for response.
>>>> 
>>>> On Mon, 5 Jun 2023 01:06:37 -0700 (PDT)
>>>> Jan  wrote:
>>>> 
>>>> > Repeating Justin's consideration: one of my (and from colleagues I
>>>> > discuss the topic with) major issues with current error handling is
>>>> > the repetition of identical code. Your proposal still requires `when
>>>> > err handle ...` at every statement.
>>>> 
>>>> Yes, correct.
>>>> The idea is not to minimise repetition on error handling statement, the
>>>> "if err != nil", but to minimise repetitive error handling body by
>>>> grouping it into single handle based on the context of error.
>>> 
>>> Oh, I see. I have to say I wish one didn't need to do the `if err != nil` 
>>> all the time either, when the handling is the same :) But if that is your 
>>> proposal it makes sense.
>>> 
>>>> > It also doesn't allow for nested
>>>> > call of functions that return errors -- e.g: `f(g(h(x)))`, where `h`,
>>>> > `g`, and `f` can return errors (all presumably handled the same way).
>>>> > 
>>>> 
>>>> I am not sure I understand this, but should not each function handle it
>>>> on its own?
>>>> Or should not `f` being called if `g` return an error?
>>>> If its later, yes, it does not allow for nested handling of error.
>>> 
>>> Sorry I was not clear. What I mean is, with current error handling would be 
>>> written like:
>>> 
>>> hValue, err := h(x)
>>> if err != nil {...}
>>> gValue, err := g(hValue)
>>> if err != nil {...}
>>> fValue, err := f(gValue)
>>> if err != nil {...}
>>> 
>>> Where all of the `if err` are identical.
>>> 
>>> Let me provide a more concrete example: I'm working on an ML framework 
>>> (github.com/gomlx/gomlx), and I'm writing computational graphs (that get 
>>> just-in-time compiled for speed/GPU support) like:
>>> 
>>> func EuclideanDistance(x, y *Node) *Node {
>>>return Sqrt(ReduceAllSum(Square(Sub(x, y
>>> }
>>> 
>>> Where each of these functions could return an error (if shapes are 
>>> incompatible). If an error is raised (they hold a stack trace), I want them 
>>> simply propagated upwards. Error handling in the middle of these math 
>>> expressions (some are larger) get in the way of the math being expressed.
>>> 
&

Re: [go-nuts] Re: [RFC] Yet another proposal for Go2 error handling

[Robert Engels]

Seems  the easiest way to address that is with a DSL and a code generator. 

> On Jun 18, 2023, at 9:47 AM, Jan Pfeifer  wrote:
> 
> 
> hi Shulhan, I see your points, thanks for the reply. Let me comment on them 
> below:
> 
>> On Sat, Jun 17, 2023 at 9:21 AM Shulhan  wrote:
>> Hi Jan, thanks for response.
>> 
>> On Mon, 5 Jun 2023 01:06:37 -0700 (PDT)
>> Jan  wrote:
>> 
>> > Repeating Justin's consideration: one of my (and from colleagues I
>> > discuss the topic with) major issues with current error handling is
>> > the repetition of identical code. Your proposal still requires `when
>> > err handle ...` at every statement.
>> 
>> Yes, correct.
>> The idea is not to minimise repetition on error handling statement, the
>> "if err != nil", but to minimise repetitive error handling body by
>> grouping it into single handle based on the context of error.
> 
> Oh, I see. I have to say I wish one didn't need to do the `if err != nil` all 
> the time either, when the handling is the same :) But if that is your 
> proposal it makes sense.
> 
>> > It also doesn't allow for nested
>> > call of functions that return errors -- e.g: `f(g(h(x)))`, where `h`,
>> > `g`, and `f` can return errors (all presumably handled the same way).
>> > 
>> 
>> I am not sure I understand this, but should not each function handle it
>> on its own?
>> Or should not `f` being called if `g` return an error?
>> If its later, yes, it does not allow for nested handling of error.
> 
> Sorry I was not clear. What I mean is, with current error handling would be 
> written like:
> 
> hValue, err := h(x)
> if err != nil {...}
> gValue, err := g(hValue)
> if err != nil {...}
> fValue, err := f(gValue)
> if err != nil {...}
> 
> Where all of the `if err` are identical.
> 
> Let me provide a more concrete example: I'm working on an ML framework 
> (github.com/gomlx/gomlx), and I'm writing computational graphs (that get 
> just-in-time compiled for speed/GPU support) like:
> 
> func EuclideanDistance(x, y *Node) *Node {
>return Sqrt(ReduceAllSum(Square(Sub(x, y
> }
> 
> Where each of these functions could return an error (if shapes are 
> incompatible). If an error is raised (they hold a stack trace), I want them 
> simply propagated upwards. Error handling in the middle of these math 
> expressions (some are larger) get in the way of the math being expressed.
> 
> cheers 
> Jan
> 
> 
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAE%3D7LsUQfPLP1w0s9EN3XZSsoproEStkQHRb-s8pcdTgmpbzTw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6115E519-3E68-4DD2-877D-66EB54ECA26F%40ix.netcom.com.

Re: [go-nuts] is running interactive or not

[Robert Engels]

No need to even think about this. Look at ssh and mimic their practices. It has 
been thoroughly audited. 

> On Jun 10, 2023, at 5:35 AM, quin...@gmail.com  wrote:
> 
> there has been much discussion in the past about programs that modify their 
> behaviour depending on what stdout is; 
> http://harmful.cat-v.org/cat-v/unix_prog_design.pdf
> 
> i do to want to start a war, just suggest a different approach is available.
> 
> my suggestion would be to always expect a password from standard input and to 
> do document this, suggesting, for non-interactive use people could do 
> something like:
> 
>  echo password | application
> 
> using environment variables or passing passwords on the command line are 
> rather leaky.
> 
> -Steve
> 
> 
> 
>> On Friday, 9 June 2023 at 00:52:58 UTC+1 Rich wrote:
>> Thank you Cris and Kurtis -- For this project I am going with the switch 
>> option -- but I have other programs that I am going to replace the os.Getpid 
>> and os.Getppid trick with go-isatty.
>> 
>> 
>> 
>>> On Thursday, June 8, 2023 at 3:22:29 PM UTC-4 Chris Burkert wrote:
>>> Hi, there are cases when this does not work. I tend to use a flag like 
>>> -batch or -noninteractive to trigger the correct behavior from within 
>>> scripts. Less magic, more control.
>>> 
>>> Rich  schrieb am Do. 8. Juni 2023 um 20:19:
 Hi, 
 
 I have a program I am writing that stops and asks the user for input, in 
 this case it's a passphrase used to encrypt output.
 
  I want the program to also be able to be used in a script, and if in a 
 script use a predefined value as the passphrase. What I'd like to know is 
 how to detect if running in a script or not.  I've tried something like 
 this:
 runPid := os.Getpid()
 parPid := os.Getppid()
 val := runPid - parPid //normally I check if runPid is > parPid in my code.
 if val < 20 {
   fmt.Println("Not running in a script")
 }
 This works, but only if the script ran quickly. Wondering if there is a 
 better way to do this?
 
>>> 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to golang-nuts...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/golang-nuts/c8ae1be5-5a6b-45af-9249-ccdb02283d97n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/b5285e0c-51bf-4ec5-8682-70b5a324c32bn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1623BF2C-4FB9-458A-813B-5FD80A89D443%40ix.netcom.com.

Re: [go-nuts] Can we further optimize the scheduling order of goroutines in sync.Mutex?

[Robert Engels]

That is implicit since the list of waiters is ordered. 

> On Jun 5, 2023, at 7:08 AM, fliter  wrote:
> 
> Thanks for your answer. But I wonder if the elements behind the queue may 
> wait for a very long time? Can we maintain a waiting time after entering the 
> queue for each coroutine, and acquire locks from high to low
> 
> Thanks again.
> 
> 在2023年6月4日星期日 UTC+8 01:07:52 写道：
>> On Sat, Jun 3, 2023 at 12:35 AM fliter  wrote: 
>> > 
>> > 
>> > In sync/mutex.go, there is a comment like the following: 
>> > 
>> > ```go 
>> > // Mutex fairness. 
>> > // 
>> > // Mutex can be in 2 modes of operations: normal and starvation. 
>> > // In normal mode waiters are queued in FIFO order, but a woken up waiter 
>> > // does not own the mutex and competes with new arriving goroutines over 
>> > // the ownership. New arriving goroutines have an advantage -- they are 
>> > // already running on CPU and there can be lots of them, so a woken up 
>> > // waiter has good chances of losing. In such case it is queued at front 
>> > // of the wait queue. If a waiter fails to acquire the mutex for more than 
>> > 1ms, 
>> > // it switches mutex to the starvation mode. 
>> > ``` 
>> > 
>> > 
>> > I wonder if the waiter here refers to the goroutine at the head of the 
>> > queue, or any goroutine in the queue? 
>> 
>> In that comment "waiter" refers to the goroutine at the head of the 
>> queue. When a mutex is unlocked, it wakes up the first waiter, if 
>> any. 
>> 
>> Ian 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/8a9dd408-a4cc-45e6-b806-e8a25cb4e309n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B0F047D3-ED20-4FA2-83D0-8457FA992C12%40ix.netcom.com.

Re: [go-nuts] Re: Lots of retained free spans

[Robert Engels]

Go GC does not use a compacting/copying collector so depending on your 
allocation patterns you could see this behavior. But you’re stating > 70% free 
spade available. 

I.e It may be that these are holes that cannot be released to the OS. 

> On May 29, 2023, at 10:08 AM, Gleb Petrovichev  
> wrote:
> 
> I'm having the exactly same problem now. Did you by any chance able to 
> figure it out? I'm compiling with go1.19.8 though. Thanks
>> On Saturday, January 29, 2022 at 2:02:18 AM UTC+1 Eric Hubbard wrote:
>> I've been trying to root cause an OOM condition.  My process is running 
>> within a cgroup with a 4gb limit.  Occasionally that limit gets hit.  I 
>> added a cgroup listener to watch for memory usage and create a pprof and 
>> core dump.
>> 
>> The pprof shows just a few hundred megs of "in-use" memory, however when I 
>> open the core dump with viewcore I'm seeing around 3.3gb of free spans that 
>> are being retained.  "kept for reuse by Go"..
>> 
>> I've got GOGC set to 50 -- but my understanding is that just controls when a 
>> GC kicks off -- not when memory will be returned to the OS.   
>> 
>> Is there some sort of behavior a program can do to create this type of 
>> situation?  or a way to give a hint to the GC  .. "hey... give it back!" :)  
>>  
>> 
>> Thanks for any ideas or thoughts!
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e7c467fa-d78a-4b1c-b14a-18e0d028810bn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/677A90D0-D62B-4DF3-83FA-AF6A5D258BDD%40ix.netcom.com.

Re: [go-nuts] Re: HTTP client timeout

[Robert Engels]

I mean you can use curl to simulate the 1000’s of clients (depending on 
protocol dependencies)

> On May 5, 2023, at 9:23 AM, envee  wrote:
> 
> Hi Robert, Yes I have tried HTTP request via curl. That yields a response in 
> about 200ms.
> 
>> On Friday, 5 May 2023 at 22:56:58 UTC+10 Robert Engels wrote:
>> Have you tried using curl as the client in order to narrow down the problem?
>> 
>>>> On May 5, 2023, at 7:45 AM, envee  wrote:
>>>> 
>>> An update. 
>> 
>>> The VM on which I am running has 40 vCPUs.
>>> I changed my GOMAXPROCS to a lower value of 8.
>>> And also changed my client to use 90 HTTP/2 connections towards the HTTP/2 
>>> server.
>>> With this setup, I can now achieve about 9000 Txns per second.
>>> However, if I go any higher than that, my client starts throwing the 
>>> "context deadline exceeded" errors. I am sure that the responses are 
>>> received by the client well within the timeout of 7 seconds.
>>> When I observe the CPU utilization of my client process, it only uses about 
>>> 300% vCPU i.e. 3 out of 40 vCPUs.
>>> Still, I don't understand why I cannot achieve higher throughput than 9000 
>>> per second.
>>> Here is a sample output from the atop utility. As seen below, only 257% 
>>> shows as CPU utilization.
>>> 
>>> Is this because goroutines are not getting context switched onto a logical 
>>> processor or thread. I thought if there is so much CPU available, then 
>>> context switching should be very fast and not an issue at all.
>>> Thus leading to timeouts ?
>>> 
>>> ATOP - myserver  2023/05/05  22:16:19   
>>>yP   
>>>10s elapsed
>>> PRC | sys6.63s | user  21.56s |  | #proc543 | #trun 
>>>  4 | #tslpi   639 |   | #tslpu 0 | #zombie0 | clones
>>> 11 |  | no  procacct |
>>> CPU | sys  61% | user213% | irq   4% | idle   3724% | wait  
>>> 0% | steal 0% |  guest 0% |  | ipc notavail | cycl 
>>> unknown | curf 3.00GHz | curscal   ?% |
>>> CPL | avg12.28 | avg52.40 |  | avg15   2.45 |   
>>>|  |  csw   656705 | intr  515996 |  |   
>>>| numcpu40 |  |
>>> MEM | tot   503.7G | free  468.6G | cache  26.9G | dirty   0.1M | buff
>>> 1.0G | slab1.4G |  slrec   1.2G | shmem   4.1G | shrss 117.0M | vmbal   
>>> 0.0M | hptot   0.0M | hpuse   0.0M |
>>> SWP | tot 4.0G | free4.0G |  |  |   
>>>|  |   |  |  |   
>>>| vmcom   6.0G | vmlim 255.8G |
>>> LVM | g_vd0-lv_var | busy  0% | read   0 |  | write
>>> 158 | KiB/r  0 |  KiB/w  4 | MBr/s0.0 | MBw/s0.1 |  
>>> | avq 5.00 | avio 0.01 ms |
>>> DSK |  sda | busy  0% | read   0 |  | write
>>> 122 | KiB/r  0 |  KiB/w  5 | MBr/s0.0 | MBw/s0.1 |  
>>> | avq 2.00 | avio 0.01 ms |
>>> NET | transport| tcpi  181070 | tcpo  289884 | udpi  18 | udpo  
>>> 18 | tcpao  0 |  tcppo 26 | tcprs556 | tcpie  0 | tcpor 
>>>  1 | udpnp  0 | udpie  0 |
>>> NET | network  | ipi   181092 | ipo   201839 |  | ipfrw 
>>>  0 | deliv 181091 |   |  |  |   
>>>| icmpi  0 | icmpo  0 |
>>> NET | bond1 0% | pcki  181057 | pcko  290688 | sp   20 Gbps | si   16 
>>> Mbps | so  190 Mbps |  coll   0 | mlti   0 | erri   0 | erro
>>>0 | drpi   0 | drpo   0 |
>>> NET | net2  0% | pcki  181056 | pcko  290688 | sp   20 Gbps | si   16 
>>> Mbps | so  190 Mbps |  coll   0 | mlti   0 | erri   0 | erro
>>>0 | drpi   0 | drpo   0 |
>>> 
>>>   PID TID   SYSCPUUSRCPUVGROWRGROW  
>>>   RUID   EUID   STEXCTHR   S
>>> CPUNRCPU   CMD 1/1
>>> 40753   -4.08s21.52s   0K   -11.7M  
>>>   mtxmtx--  - 15   S   
>>> 20   257%   5gclient
>>> 40753  

Re: [go-nuts] Re: HTTP client timeout

[Robert Engels]

Have you tried using curl as the client in order to narrow down the problem?

> On May 5, 2023, at 7:45 AM, envee  wrote:
> 
> An update. 
> The VM on which I am running has 40 vCPUs.
> I changed my GOMAXPROCS to a lower value of 8.
> And also changed my client to use 90 HTTP/2 connections towards the HTTP/2 
> server.
> With this setup, I can now achieve about 9000 Txns per second.
> However, if I go any higher than that, my client starts throwing the "context 
> deadline exceeded" errors. I am sure that the responses are received by the 
> client well within the timeout of 7 seconds.
> When I observe the CPU utilization of my client process, it only uses about 
> 300% vCPU i.e. 3 out of 40 vCPUs.
> Still, I don't understand why I cannot achieve higher throughput than 9000 
> per second.
> Here is a sample output from the atop utility. As seen below, only 257% shows 
> as CPU utilization.
> 
> Is this because goroutines are not getting context switched onto a logical 
> processor or thread. I thought if there is so much CPU available, then 
> context switching should be very fast and not an issue at all.
> Thus leading to timeouts ?
> 
> ATOP - myserver  2023/05/05  22:16:19 
>  yP   
>10s elapsed
> PRC | sys6.63s | user  21.56s |  | #proc543 | #trun  
> 4 | #tslpi   639 |   | #tslpu 0 | #zombie0 | clones11 
> |  | no  procacct |
> CPU | sys  61% | user213% | irq   4% | idle   3724% | wait  
> 0% | steal 0% |  guest 0% |  | ipc notavail | cycl 
> unknown | curf 3.00GHz | curscal   ?% |
> CPL | avg12.28 | avg52.40 |  | avg15   2.45 | 
>  |  |  csw   656705 | intr  515996 |  |  
> | numcpu40 |  |
> MEM | tot   503.7G | free  468.6G | cache  26.9G | dirty   0.1M | buff
> 1.0G | slab1.4G |  slrec   1.2G | shmem   4.1G | shrss 117.0M | vmbal   
> 0.0M | hptot   0.0M | hpuse   0.0M |
> SWP | tot 4.0G | free4.0G |  |  | 
>  |  |   |  |  |  
> | vmcom   6.0G | vmlim 255.8G |
> LVM | g_vd0-lv_var | busy  0% | read   0 |  | write
> 158 | KiB/r  0 |  KiB/w  4 | MBr/s0.0 | MBw/s0.1 |
>   | avq 5.00 | avio 0.01 ms |
> DSK |  sda | busy  0% | read   0 |  | write
> 122 | KiB/r  0 |  KiB/w  5 | MBr/s0.0 | MBw/s0.1 |
>   | avq 2.00 | avio 0.01 ms |
> NET | transport| tcpi  181070 | tcpo  289884 | udpi  18 | udpo  
> 18 | tcpao  0 |  tcppo 26 | tcprs556 | tcpie  0 | tcpor  
> 1 | udpnp  0 | udpie  0 |
> NET | network  | ipi   181092 | ipo   201839 |  | ipfrw  
> 0 | deliv 181091 |   |  |  |  
> | icmpi  0 | icmpo  0 |
> NET | bond1 0% | pcki  181057 | pcko  290688 | sp   20 Gbps | si   16 
> Mbps | so  190 Mbps |  coll   0 | mlti   0 | erri   0 | erro  
>  0 | drpi   0 | drpo   0 |
> NET | net2  0% | pcki  181056 | pcko  290688 | sp   20 Gbps | si   16 
> Mbps | so  190 Mbps |  coll   0 | mlti   0 | erri   0 | erro  
>  0 | drpi   0 | drpo   0 |
> 
>   PID TID   SYSCPUUSRCPUVGROWRGROW
> RUID   EUID   STEXCTHR   SCPUNR   
>  CPU   CMD 1/1
> 40753   -4.08s21.52s   0K   -11.7M
> mtxmtx--  - 15   S   20   
> 257%   5gclient
> 40753   407530.00s 0.00s   0K   -11.7M
> mtxmtx--  -  1   S   20   
>   0%   5gclient  
> 40753   407540.00s 0.00s   0K   -11.7M
> mtxmtx--  -  1   S5   
>   0%   5gclient  
> 40753   407550.00s 0.00s   0K   -11.7M
> mtxmtx--  -  1   S3   
>   0%   5gclient  
> 40753   407560.00s 0.00s   0K   -11.7M
> mtxmtx--  -  1   S   21   
>   0%   5gclient  
> 40753   407570.45s 2.35s   0K   -11.7M
> mtxmtx--  -  1   S6   
>  28%   5gclient  
> 40753   407580.44s 2.31s   0K   -11.7M
> mtxmtx--  -  1  

Re: [go-nuts] Re: Short Variable Declarations Are Oversold

[Robert Engels]

If you use Go’s “accept interfaces return concrete types” the dev/ide knows to 
declare the concrete type. 

Type inference can actually cause more issues during refactorings - not 
alleviate them. 

Personally, I find types make the code easier to read. But when I write ago I 
don’t use them. I find when I review old code it is a bit harder.

In Java I think it is less of an issue (on refactoring) because you normally 
use the interface and due to implements you have more safety. 

> On Apr 24, 2023, at 2:14 AM, a2800276  wrote:
> 
> 
> import “os”
> func main() {
>   file, _ := os.Open(os.Args[1])
>   myfunc(file)
> }
> 
> func myfunc(file ???) {
> }
> 
> What type should I use to declare “file” in the parameter list for myfunc()? 
> 
> This argument doesn't seem logical to me. If I fully declare `file` in this 
> case, I also have to know which type to declare it as:
> 
>  var file ???;
> file, _ = os.Open(...)
> 
> This will also be a "distraction and waste time". I'm all for explicit 
> declarations in cases where they add clarity, but in idiomatic cases, I don't 
> feel it adds anything useful.
> 
> Additionally, in cases where I am creating a new function to work with the 
> variable, surely I'm aware of more context concerning the API, e.g. in the 
> example, I'll assume `file` has `Read` and `Close`, but passing the variable 
> along, I would assume the operation on the variable will be more involved and 
> requiring reading the documentation anyway ... 
> 
>  -tim
> 
> 
> 
> 
> 
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/f6c9bafb-50a9-4e40-b091-59988417ebffn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/81837BA2-D643-4526-9C79-FAEDD91C1BBA%40ix.netcom.com.

Re: [go-nuts] Short Variable Declarations Are Oversold

[Robert Engels]

I wouldn’t have don’t it that way. I would have made := mean all variables must 
be new. And = mean any new variable requires a decoration. 

So in this case,

var x,err = means x is new and err is reused. The variable clauses are 
segregated by a ,  



> On Apr 23, 2023, at 9:07 AM, Axel Wagner  
> wrote:
> 
> 
> 
> On Sun, Apr 23, 2023 at 4:01 PM Robert Engels  wrote:
>> Personally that syntax has always bothered me with readability. It requires 
>> lots of previous knowledge in some cases. A syntax like
>> 
>> x, var y int = blah
>> 
>> Is more explicit that x is reused and y is declared. 
> 
> That specific suggestion would not work. It would mean it is ambiguous 
> whether `var x, err = fmt.Println()` is mean to be a declaration of only x, 
> or both - and note that this specific order is the more relevant, as `err` is 
> the specific variable that tends to be re-used by `:=`.
> There probably could be alternatives though. But that seems a moot point now.
> 
>  
>> 
>> Go is all about being explicit until it isn’t. 
>> 
>> 
>> 
>>> On Apr 23, 2023, at 8:28 AM, 'Axel Wagner' via golang-nuts 
>>>  wrote:
>>> 
>>> Just to nit-pick everyone: Short variable declarations are not there to 
>>> omit type information. You can do that with a regular variable declaration:
>>> https://go.dev/play/p/6XePFCh-6G2
>>> Short variable declarations exist to 1. be shorter and 2. allow you to 
>>> avoid re-declaration errors when assigning multiple variables:
>>> https://go.dev/play/p/bgbU9mTunhL
>>> So, IMO short variable declarations definitely increase readability, just 
>>> by that latter effect. Type-inference is a bonus.
>>> 
>>> On Sun, Apr 23, 2023 at 3:09 PM Jesper Louis Andersen 
>>>  wrote:
>>>> On Sun, Apr 23, 2023 at 12:31 AM jlfo...@berkeley.edu 
>>>>  wrote:
>>>>> 
>>>>> Short definitions detract from one of Go’s primary goals - readability. I 
>>>>> started using Go in the first place because I wanted a strongly typed 
>>>>> language with explicit type declarations. 
>>>> 
>>>> Your claim of readability is not held by everyone. Some people prefer 
>>>> there be no type information in a program because the type information 
>>>> "detracts from what the program is doing". Hence, it becomes rather hard 
>>>> to please everybody.
>>>> 
>>>> Short variable declarations are a poor man's type inference. In fully 
>>>> type-inferred languages, you can omit types everywhere, and the compiler 
>>>> will deduce an appropriate type for each declaration. It will typically 
>>>> pick the most general type for an expression. The type information is 
>>>> still there, but it is generated on-demand by the compiler, and programs 
>>>> which fail the type check are rejected. Haskell and OCaml are good 
>>>> examples of programming languages following this style. Yet in both 
>>>> languages, you often see type declarations sprinkled throughout the code 
>>>> base to guide the reader. You sort-of assume a certain amount of 
>>>> experience, and add types as you see fit to capture that experience. 
>>>> Often, you end up with your interfaces being type-annotated, but your 
>>>> internal code avoiding annotation.
>>>> 
>>>> The grand advantage of type inference is that the types can vary easily. 
>>>> If you change a fundamental type, the compiler will check that your change 
>>>> is sound. And you don't have to go around the code base and change every 
>>>> occurrence. That's a really nice boon.
>>>> 
>>>> We are slowly moving into a world where the compiler and the programmer 
>>>> are working on the code at the same time. You ask the compiler to fill out 
>>>> gaps in the programs you are writing. The result is that your editor can 
>>>> live-annotate the appropriate types of declarations and expressions 
>>>> because it can be lifted from the compiler. When I write OCaml, for 
>>>> instance, my editor annotates functions with types for me by adding a line 
>>>> above the function declaration in a smaller font. These lines only occur 
>>>> virtually in the buffer, and aren't present in the program file.
>>>> 
>>>> For some languages, such as Agda, the interaction is even stronger: you 
>>>> can ask the compiler to fill in parts of the program based on the types 
>>>> the

Re: [go-nuts] Short Variable Declarations Are Oversold

[Robert Engels]

To clarify, the semantics of a go statement from a language perspective changes 
based on earlier statements. That is simply weird - and hurts readability. 

> On Apr 23, 2023, at 9:01 AM, Robert Engels  wrote:
> 
> 
> Personally that syntax has always bothered me with readability. It requires 
> lots of previous knowledge in some cases. A syntax like
> 
> x, var y int = blah
> 
> Is more explicit that x is reused and y is declared. 
> 
> Go is all about being explicit until it isn’t. 
> 
> 
> 
>>> On Apr 23, 2023, at 8:28 AM, 'Axel Wagner' via golang-nuts 
>>>  wrote:
>>> 
>> 
>> Just to nit-pick everyone: Short variable declarations are not there to omit 
>> type information. You can do that with a regular variable declaration:
>> https://go.dev/play/p/6XePFCh-6G2
>> Short variable declarations exist to 1. be shorter and 2. allow you to avoid 
>> re-declaration errors when assigning multiple variables:
>> https://go.dev/play/p/bgbU9mTunhL
>> So, IMO short variable declarations definitely increase readability, just by 
>> that latter effect. Type-inference is a bonus.
>> 
>>> On Sun, Apr 23, 2023 at 3:09 PM Jesper Louis Andersen 
>>>  wrote:
>>>> On Sun, Apr 23, 2023 at 12:31 AM jlfo...@berkeley.edu 
>>>>  wrote:
>>>> 
>>>> Short definitions detract from one of Go’s primary goals - readability. I 
>>>> started using Go in the first place because I wanted a strongly typed 
>>>> language with explicit type declarations. 
>>>> 
>>> 
>>> Your claim of readability is not held by everyone. Some people prefer there 
>>> be no type information in a program because the type information "detracts 
>>> from what the program is doing". Hence, it becomes rather hard to please 
>>> everybody.
>>> 
>>> Short variable declarations are a poor man's type inference. In fully 
>>> type-inferred languages, you can omit types everywhere, and the compiler 
>>> will deduce an appropriate type for each declaration. It will typically 
>>> pick the most general type for an expression. The type information is still 
>>> there, but it is generated on-demand by the compiler, and programs which 
>>> fail the type check are rejected. Haskell and OCaml are good examples of 
>>> programming languages following this style. Yet in both languages, you 
>>> often see type declarations sprinkled throughout the code base to guide the 
>>> reader. You sort-of assume a certain amount of experience, and add types as 
>>> you see fit to capture that experience. Often, you end up with your 
>>> interfaces being type-annotated, but your internal code avoiding annotation.
>>> 
>>> The grand advantage of type inference is that the types can vary easily. If 
>>> you change a fundamental type, the compiler will check that your change is 
>>> sound. And you don't have to go around the code base and change every 
>>> occurrence. That's a really nice boon.
>>> 
>>> We are slowly moving into a world where the compiler and the programmer are 
>>> working on the code at the same time. You ask the compiler to fill out gaps 
>>> in the programs you are writing. The result is that your editor can 
>>> live-annotate the appropriate types of declarations and expressions because 
>>> it can be lifted from the compiler. When I write OCaml, for instance, my 
>>> editor annotates functions with types for me by adding a line above the 
>>> function declaration in a smaller font. These lines only occur virtually in 
>>> the buffer, and aren't present in the program file.
>>> 
>>> For some languages, such as Agda, the interaction is even stronger: you can 
>>> ask the compiler to fill in parts of the program based on the types they 
>>> have. That is, types and terms coalesce and there is no stratification 
>>> between them. Writing a term makes the compiler deduce the type. Writing a 
>>> type makes the compiler deduce and fill in the term. Coming strong into 
>>> this are large language models from machine learning. You can fill in lots 
>>> of gaps in programs via LLMs. Programming often contains a lot of 
>>> janitorial tasks around a computational kernel and LLMs can accelerate the 
>>> janitor. In the future, I hope someone takes an LLM and starts exploiting 
>>> type information. I have a hunch it's going to be far more effective for 
>>> languages which have static type systems (inferred or not) because there's 
>>> a much riche

Re: [go-nuts] Short Variable Declarations Are Oversold

[Robert Engels]

Personally that syntax has always bothered me with readability. It requires 
lots of previous knowledge in some cases. A syntax like

x, var y int = blah

Is more explicit that x is reused and y is declared. 

Go is all about being explicit until it isn’t. 



> On Apr 23, 2023, at 8:28 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> Just to nit-pick everyone: Short variable declarations are not there to omit 
> type information. You can do that with a regular variable declaration:
> https://go.dev/play/p/6XePFCh-6G2
> Short variable declarations exist to 1. be shorter and 2. allow you to avoid 
> re-declaration errors when assigning multiple variables:
> https://go.dev/play/p/bgbU9mTunhL
> So, IMO short variable declarations definitely increase readability, just by 
> that latter effect. Type-inference is a bonus.
> 
>> On Sun, Apr 23, 2023 at 3:09 PM Jesper Louis Andersen 
>>  wrote:
>>> On Sun, Apr 23, 2023 at 12:31 AM jlfo...@berkeley.edu 
>>>  wrote:
>>> 
>>> Short definitions detract from one of Go’s primary goals - readability. I 
>>> started using Go in the first place because I wanted a strongly typed 
>>> language with explicit type declarations. 
>>> 
>> 
>> Your claim of readability is not held by everyone. Some people prefer there 
>> be no type information in a program because the type information "detracts 
>> from what the program is doing". Hence, it becomes rather hard to please 
>> everybody.
>> 
>> Short variable declarations are a poor man's type inference. In fully 
>> type-inferred languages, you can omit types everywhere, and the compiler 
>> will deduce an appropriate type for each declaration. It will typically pick 
>> the most general type for an expression. The type information is still 
>> there, but it is generated on-demand by the compiler, and programs which 
>> fail the type check are rejected. Haskell and OCaml are good examples of 
>> programming languages following this style. Yet in both languages, you often 
>> see type declarations sprinkled throughout the code base to guide the 
>> reader. You sort-of assume a certain amount of experience, and add types as 
>> you see fit to capture that experience. Often, you end up with your 
>> interfaces being type-annotated, but your internal code avoiding annotation.
>> 
>> The grand advantage of type inference is that the types can vary easily. If 
>> you change a fundamental type, the compiler will check that your change is 
>> sound. And you don't have to go around the code base and change every 
>> occurrence. That's a really nice boon.
>> 
>> We are slowly moving into a world where the compiler and the programmer are 
>> working on the code at the same time. You ask the compiler to fill out gaps 
>> in the programs you are writing. The result is that your editor can 
>> live-annotate the appropriate types of declarations and expressions because 
>> it can be lifted from the compiler. When I write OCaml, for instance, my 
>> editor annotates functions with types for me by adding a line above the 
>> function declaration in a smaller font. These lines only occur virtually in 
>> the buffer, and aren't present in the program file.
>> 
>> For some languages, such as Agda, the interaction is even stronger: you can 
>> ask the compiler to fill in parts of the program based on the types they 
>> have. That is, types and terms coalesce and there is no stratification 
>> between them. Writing a term makes the compiler deduce the type. Writing a 
>> type makes the compiler deduce and fill in the term. Coming strong into this 
>> are large language models from machine learning. You can fill in lots of 
>> gaps in programs via LLMs. Programming often contains a lot of janitorial 
>> tasks around a computational kernel and LLMs can accelerate the janitor. In 
>> the future, I hope someone takes an LLM and starts exploiting type 
>> information. I have a hunch it's going to be far more effective for 
>> languages which have static type systems (inferred or not) because there's a 
>> much richer set of information you can exploit.
>> 
>> 
>> 
>> -- 
>> J.
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAGrdgiVd3BMOKE6ohRbwTmC_AhSY3Zht4LxK%3DFQjqj_YocoZAg%40mail.gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFtaM2KMPxRYdaFS79O0vf91RPzQBHwHa2CLJWB6r5DJQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 

Re: [go-nuts] Improving IPC latency?

[Robert Engels]

Linux context switch is 6-7 usec. Usually it is more cache invalidation that 
causes the whole code to run slow. Usually spinning is the only solution and 
core/cpu isolation. 

> On Apr 9, 2023, at 1:33 PM, TH  wrote:
> 
> Hey,
> 
> I'm building a software which handles hundred-ish concurrent network 
> communications. I want to build some other parts of core logic in to a 
> separate process. I've tried to use UDS and pipes, but they all give me ~25µs 
> (RTT 50µs) latency and sometimes on idle 300µs (RTT 600µs).
> 
> The 25µs is not a big problem, but the big variance in latency is an 
> uncovenience. Are there any methods to synchronize between two processes with 
> lower latency jitter? Throughput is not an issue, as I can always use shared 
> memory to pass big data. But I need a low latency way to signal another 
> process that new data is in the shared memory.
> 
> I can get it to less than 5µs with spinlock on shared memory, but that's not 
> a solution as CPU is 100%. Are there any primitives that we could use e.g. 
> shared futex? Or the context switch is typically that heavy?
> 
> Thanks
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/4e58d2c0-e097-49a3-afe2-af4ddac214f4n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F106DF3D-371A-4F95-BD5C-D38AFE47052A%40ix.netcom.com.

Re: [go-nuts] Need help: Calling panic() is corrupting system stack

[robert engels]

Unfortunately, I am on OSX - and Go doesn’t appear to generate a core dump on panic despite the GOTRACEBACK setting.That being said, the stack-traces look ok to me. See attached.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9E0182AE-9759-4371-8DE8-2B837438F9B9%40ix.netcom.com.


test.out.txt
Description: application/applefile
On Apr 3, 2023, at 11:59 PM, mariappan balraj <mariappan.bal...@gmail.com> wrote:Hi Robert,I have published the test case on github. Please find the links. Please let me know if you need any information. I am ready to provide and support.https://github.com/MariappanBalraj/test_cgo_panichttps://github.com/MariappanBalraj/test_cgo_panic.githttps://github.com/MariappanBalraj/test_cgo_panic/archive/refs/heads/main.zipBest RegardsMariappanOn Mon, Apr 3, 2023 at 9:50 AM Robert Engels <reng...@ix.netcom.com> wrote:I would start by publishing a small reproducible test case on github that fully compiles that demonstrates the issue.  On Apr 2, 2023, at 10:59 PM, mariappan balraj <mariappan.bal...@gmail.com> wrote:Hello Go Experts,It is good if someone can help on this?Best RegardsMariappanOn Sat, Apr 1, 2023 at 7:45 AM mariappan balraj <mariappan.bal...@gmail.com> wrote:Hello Go experts,Could someone please help to resolve this issue?Best RegardsMariappanOn Thu, Mar 30, 2023 at 2:52 PM mariappan balraj <mariappan.bal...@gmail.com> wrote:Hello Go Experts,When panic() is called from Go function, in the below call sequence, runtime.unwindm() [which is defer function of  runtime.cgocallbackg1] is called. This function is unwinding the system stack. Later, as part of the function addOneOpenDeferFrame(), systemstack() is called to run a function in the system stack. This will use the stack which is allocated for C function calls. This makes stack unwinding impossible in case of panic() is called. Can someone please help me to fix this issue?(dlv) bt 0  0x004054e6 in runtime.unwindm    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:326 1  0x00405446 in runtime.cgocallbackg1.func3    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:304 2  0x004340c8 in runtime.deferCallSave    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:796 3  0x00433fa5 in runtime.runOpenDeferFrame    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:769 4  0x00434332 in runtime.gopanic    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:884 5  0x004642a7 in main.Test4    at ./export.go:7    at ./export.go:7 6  0x0046431c in _cgoexp_78b81bbf688e_Test4    at _cgo_gotypes.go:61 7  0x0040535b in runtime.cgocallbackg1    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:315 8  0x00405079 in runtime.cgocallbackg    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:234 9  0x00461b0f in runtime.cgocallbackg    at :110  0x0045f3f4 in runtime.cgocallback    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:99811  0x004641bd in crosscall2    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgo/asm_amd64.s:3012  0x00464386 in C.Test4    at /tmp/go-build/_cgo_export.c:3313  0x00464432 in C.test1    at ./hello.go:914  0x0046444d in C.test2    at ./hello.go:1415  0x00464468 in C.test315  0x00464468 in C.test3    at ./hello.go:1916  0x004644a7 in C._cgo_78b81bbf688e_Cfunc_test3    at /tmp/go-build/cgo-gcc-prolog:4917  0x0045f2e4 in runtime.asmcgocall    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:84818  0x0046448a in C._cgo_78b81bbf688e_Cfunc_test3    at /tmp/go-build/cgo-gcc-prolog:4419  0x00404f0a in runtime.cgocall    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:16720  0x00464245 in main._Cfunc_test3    at _cgo_gotypes.go:3921  0x004642d7 in main.main    at ./hello.go:3322  0x00437073 in runtime.main    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/proc.go:25023  0x0045f5e1 in runtime.goexit    at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:1598254 func cgocallbackg1(fn, frame unsafe.Pointer, ctxt uintptr) {304         defer unwindm()326 func unwindm(restore *bool) {327         if *restore {328                 // Restore sp saved by cgocallback during329                 // unwind of g's stack (see comment at top of file).330                 mp := acquirem()331                 sched := 332                 sched.sp = *(*uintptr)(unsafe.Pointer(sched.sp + alignUp(sys.MinFrameSize, sys.StackAlign)))(dlv) > runtime.addOneOpenDeferFrame() /home/soomohan/mbalraj/

Re: [go-nuts] Interesting "select" examples

[Robert Engels]

Interesting - that page has “ Thread stacks are allocated in shared memory, 
making it valid to pass pointers to stack variables between threads and procs.”

That seems very difficult to pull off safely.

> On Apr 3, 2023, at 9:46 PM, Skip Tavakkolian  
> wrote:
> 
> On Mon, Apr 3, 2023 at 5:22 PM Nigel Tao  wrote:
>> 
>>> On Tue, Apr 4, 2023 at 7:56 AM Skip Tavakkolian
>>>  wrote:
>>> mux9p, Fazlul Shahriar's port of Russ' 9pserve (plan9port)
>>> https://github.com/fhs/mux9p/search?q=clientIO
>>> 
>>> ...
>>> 
>>>select {
>>>case v, ok := <- counts:
>>>// collect samples
>>>case reporting <- Stats{ stats }:
>>>case <-timer:
>>>   // calculate stats from samples
>>> case cmd, ok := <-commands:
>>>   // reset counters, restart worker, exit, etc.
>>>}
>> 
>> I'd have to study mux9p for longer, but its select chooses between two
>> receives, so it could possibly collapse to a single heterogenous
>> channel. Indeed, both its channels are already heterogenous in some
>> sense. Both its processTx and processRx methods say "switch
>> m.tx.Type".
> 
> One side serializes the Fcall to 9P packet and the other decodes the
> packet into Fcall (there is a check m.tx.Type+1 == m.rx.Type;  by 9P
> convention).
> 
>> 
>> Your second suggestion, mixing both sends (on the reporting channel)
>> and receives, is interesting. Would it be possible to move the
>> `reporting <- Stats{ stats }` case inside the `case <-timer: //
>> calculate stats from samples` body, though?
> 
> Yes, but you'd have to have another select if you don't want to block
> on reporting. I have an old piece of code that does that:
> https://github.com/9nut/tcpmeter/search?q=Dispatch
> 
> I don't know if you're looking for ideas on potential API's for C
> implementation, but Plan 9's (and P9P) C API for chan/select is easy
> to understand and use (https://9p.io/magic/man2html/2/thread).
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJSxfm%2BheUJBAWd3fub-9JD2%3DovPR_oqfxWT0Dpnpg8aq5jvFw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/267AC316-F894-4DCB-90E1-465E9B53AC69%40ix.netcom.com.

Re: [go-nuts] Interesting "select" examples

[Robert Engels]

Nothing concrete but I envision a system where the super set is all the request 
channels - when a request is received a routine is spawned/called with the 
related request response and status channels - a subset of all channels.  

Contrived but just thinking out loud. 

> On Apr 3, 2023, at 8:28 PM, Nigel Tao  wrote:
> On Tue, Apr 4, 2023 at 11:22 AM Robert Engels  wrote:
>> the processing often involves subsequent stages operating on a subset of the 
>> original channels based on the earlier.
> 
> I get that, in theory. Rob's pause channel is a simple example of
> that. Do you have any other concrete examples?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8A45F1D9-E1B7-4838-913B-57698B49259C%40ix.netcom.com.

Re: [go-nuts] Interesting "select" examples

[Robert Engels]

Tbh I’m not sure but I am uncertain how you would get a system with 1000+ 
independent channels coalesced into a single channel - seems that would become 
highly contended/bottleneck.  More importantly, I think the processing often 
involves subsequent stages operating on a subset of the original channels based 
on the earlier. If the channels were already merged into a heterogeneous queue 
seems that would be more difficult. 

Also, with descriptors/Go channels each can be independently buffered. Not 
saying your C++ impl couldn’t allow similar behavior. 



> On Apr 3, 2023, at 7:49 PM, Nigel Tao  wrote:
> 
> On Tue, Apr 4, 2023 at 10:13 AM Robert Engels  wrote:
>> I think it is as simple as looking at kselect/poll in posix. Each descriptor 
>> can be viewed as an independent channel of info/objects.
>> 
>> Select with Go channels works similarly and has similar usefulness.
> 
> But would any problems be easier/harder if kselect/poll instead
> presented a 'channel' of ready file descriptors - a "single,
> heterogenous input channel"?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EA96A9DA-E5B6-4AD4-A2D9-538A39DDFE5E%40ix.netcom.com.

Re: [go-nuts] Interesting "select" examples

[Robert Engels]

I think it is as simple as looking at kselect/poll in posix. Each descriptor 
can be viewed as an independent channel of info/objects. 

Select with Go channels works similarly and has similar usefulness. 

> On Apr 3, 2023, at 7:00 PM, Nigel Tao  wrote:
> 
> On Mon, Apr 3, 2023 at 8:14 PM Rob Pike  wrote:
>>select {
>>case <-ticker.C:
>>case <-pauseC:
>><-pauseC
>>}
> 
> This one is more interesting. You can't combine the ticker.C and the
> pauseC into a single heterogenous channel because you want to wait for
> the second pause event (a resumption) without dropping or buffering
> arbitrarily many other ticker events.
> 
> I take the general point that sometimes you're waiting on only a
> subset of incoming events.
> 
> For this particular code, though, I think you could structure it a
> different way. Instead of two channels with (ticker) events and
> (pause, resume) events, you could have two channels with heterogenous
> (ticker, pause) events and (resume) events. Yes, the `time.NewTicker`
> API equivalent would have to be passed a channel instead of returning
> a channel. The `case ' ': pauseC <- 1` code would have to alternate
> which channel it sent on. But we could eliminate the select.
> 
> To be clear, it's not that the code is better / simpler / cleaner if
> we eliminate selects. It's that, outside of Go, if we don't have
> selects, what becomes unworkably complicated enough that it's worth
> implementing selects.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOeFMNVZcoj8kp%3DDeaVqs55FiNcJrbPGPAUuLOLj0gUWqCRJKg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F7DC219C-7693-4A93-A801-7013E3FCD988%40ix.netcom.com.

Re: [go-nuts] Need help: Calling panic() is corrupting system stack

[Robert Engels]

I would start by publishing a small reproducible test case on github that fully 
compiles that demonstrates the issue.  

> On Apr 2, 2023, at 10:59 PM, mariappan balraj  
> wrote:
> 
> 
> Hello Go Experts,
> 
> It is good if someone can help on this?
> 
> Best Regards
> Mariappan
>> On Sat, Apr 1, 2023 at 7:45 AM mariappan balraj  
>> wrote:
>> Hello Go experts,
>> 
>> Could someone please help to resolve this issue?
>> 
>> Best Regards
>> Mariappan
>> 
>>> On Thu, Mar 30, 2023 at 2:52 PM mariappan balraj 
>>>  wrote:
>>> Hello Go Experts,
>>> 
>>> When panic() is called from Go function, in the below call sequence, 
>>> runtime.unwindm() [which is defer function of  runtime.cgocallbackg1] is 
>>> called. This function is unwinding the system stack. Later, as part of the 
>>> function addOneOpenDeferFrame(), systemstack() is called to run a function 
>>> in the system stack. This will use the stack which is allocated for C 
>>> function calls. This makes stack unwinding impossible in case of panic() is 
>>> called. Can someone please help me to fix this issue?
>>> 
>>> (dlv) bt
>>>  0  0x004054e6 in runtime.unwindm
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:326
>>>  1  0x00405446 in runtime.cgocallbackg1.func3
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:304
>>>  2  0x004340c8 in runtime.deferCallSave
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:796
>>>  3  0x00433fa5 in runtime.runOpenDeferFrame
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:769
>>>  4  0x00434332 in runtime.gopanic
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:884
>>>  5  0x004642a7 in main.Test4
>>> at ./export.go:7
>>> at ./export.go:7
>>>  6  0x0046431c in _cgoexp_78b81bbf688e_Test4
>>> at _cgo_gotypes.go:61
>>>  7  0x0040535b in runtime.cgocallbackg1
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:315
>>>  8  0x00405079 in runtime.cgocallbackg
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:234
>>>  9  0x00461b0f in runtime.cgocallbackg
>>> at :1
>>> 10  0x0045f3f4 in runtime.cgocallback
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:998
>>> 11  0x004641bd in crosscall2
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgo/asm_amd64.s:30
>>> 12  0x00464386 in C.Test4
>>> at /tmp/go-build/_cgo_export.c:33
>>> 13  0x00464432 in C.test1
>>> at ./hello.go:9
>>> 14  0x0046444d in C.test2
>>> at ./hello.go:14
>>> 15  0x00464468 in C.test3
>>> 15  0x00464468 in C.test3
>>> at ./hello.go:19
>>> 16  0x004644a7 in C._cgo_78b81bbf688e_Cfunc_test3
>>> at /tmp/go-build/cgo-gcc-prolog:49
>>> 17  0x0045f2e4 in runtime.asmcgocall
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:848
>>> 18  0x0046448a in C._cgo_78b81bbf688e_Cfunc_test3
>>> at /tmp/go-build/cgo-gcc-prolog:44
>>> 19  0x00404f0a in runtime.cgocall
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:167
>>> 20  0x00464245 in main._Cfunc_test3
>>> at _cgo_gotypes.go:39
>>> 21  0x004642d7 in main.main
>>> at ./hello.go:33
>>> 22  0x00437073 in runtime.main
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/proc.go:250
>>> 23  0x0045f5e1 in runtime.goexit
>>> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:1598
>>> 
>>> 254 func cgocallbackg1(fn, frame unsafe.Pointer, ctxt uintptr) {
>>> 304 defer unwindm()
>>> 
>>> 326 func unwindm(restore *bool) {
>>> 327 if *restore {
>>> 328 // Restore sp saved by cgocallback during
>>> 329 // unwind of g's stack (see comment at top of file).
>>> 330 mp := acquirem()
>>> 331 sched := 
>>> 332 sched.sp = *(*uintptr)(unsafe.Pointer(sched.sp + 
>>> alignUp(sys.MinFrameSize, sys.StackAlign)))
>>> 
>>> (dlv) 
>>> > runtime.addOneOpenDeferFrame() 
>>> > /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:642 (PC: 
>>> > 0x433a45)
>>> Warning: debugging optimized function
>>>637: func addOneOpenDeferFrame(gp *g, pc uintptr, sp unsafe.Pointer) {
>>>638: var prevDefer *_defer
>>>639: if sp == nil {
>>>640: prevDefer = gp._defer
>>>641: pc = prevDefer.framepc
>>> => 642: sp = unsafe.Pointer(prevDefer.sp)
>>>643: }
>>>644: systemstack(func() {
>>>645: gentraceback(pc, uintptr(sp), 0, gp, 0, nil, 
>>> 0x7fff,
>>>646: func(frame *stkframe, unused 
>>> unsafe.Pointer) bool {
>>>647: if prevDefer != nil && prevDefer.sp 
>>> == frame.sp {
>>> 

Re: [go-nuts] How to encode/decode string to binary in golang

[robert engels]

Am arbitrary byte can be encoded in 2 HEX characters - only a 2x increase in 
size not 8x.

> On Mar 13, 2023, at 2:35 PM, Volker Dobler  wrote:
> 
> On Monday, 13 March 2023 at 17:41:42 UTC+1 Van Fury wrote:
> Relating to my previous question, I have been reading but it is still not 
> clear to me what raw binary is, how is it different from
> text formatted binary (fmt.Sprintf("%b", s1s2Byte))?
> "text formated binary" takes a stream of bytes  and for each bit in
> this stream output "0" or "1" as letters, i.e. the bytes 48==0x30
> and 49==0x31. An eightfold blowup in size.
> Try understanding how _text_ is encoded as bytes.
>  
> Computers work on streams of bytes.
> The meaning/interpretation of a single byte can vary:
> Not all number can be stored in a single byte and
> text has to be encoded by bytes too.
> Please forget about "raw binary", there is no such thing.
> String variables are a stream of bytes and do not need any encoding,
> especially not something like "raw binary" that doesn't exist.
> A stream of bytes can be _printed_ (encoded) as decimal, hexadecimal,
> octal, base64 or even as bit ('0' and '1'). sometimes this is necessary
> because the underlying protocol cannot transmit possible byte values.
> When dealing with "binary" streams this cannot happen (it happens e.g.
> when using JSON to transmit data).
> You have to make clear what the output/encoding space should be
> (and why!): There is a inconsistency between wanting "binary" and
> a textual bit stream (which is not "binary" but text) and hexadecimal
> encoding (also not binary but text).
> You manoeuvred yourself in a corner by focusing on the solution
> instead of the actual problem.
> 
> V.
>  
> In my problem above I need to encode s1+s2 to raw binary before sending the 
> result to the server
> which then decodes the raw binary back to s1+s2.
> 
> 
> 
> On Mon, Mar 13, 2023 at 5:39 PM Van Fury  > wrote:
> Do you mean encoding should be
> rawdata := fmt.Print("%016d%s%s", len(s1), s1,s2) 
> or
> rawdata := fmt.Printf("%016d%s%s", len(s1), s1,s2)
> 
> 
> 
> On Monday, March 13, 2023 at 4:36:50 PM UTC+2 Alex Howarth wrote:
> You might be looking for strconv.ParseUint() 
> https://pkg.go.dev/strconv#ParseUint 
> 
> https://go.dev/play/p/qAO9LfLD41D 
> 
> On Mon, 13 Mar 2023 at 07:24, Van Fury > wrote:
> 
> Sorry I did not frame my question properly but what I would like to do is to
> encode concatenated s1 and s2 into raw binary and then decode the raw binary
> back to s1 and s2. 
> 
> 
> On Friday, March 10, 2023 at 11:36:09 PM UTC+2 Alex Howarth wrote:
> If s1 and s2 are a fixed length then you can just slice up the decoded string 
> based on the lengths. If they are of a variable length, you'll need a 
> separator in the input string to later split on when decoded (s3 := s1 + ":" 
> + <> s2 etc)?
> 
> On Fri, 10 Mar 2023 at 10:33, Van Fury > wrote:
> Hi,
> 
> I have two hexadecimal string values and would like to concatenate the two 
> strings and
> 
> encode the result to binary
> decode the resulting binary back to hexadecimal string
> I did the following but I was finding it difficult to decode the result back. 
> I ignore error check in this case.
> 
> What i did so far:
> 
> s1 := "1d28ed66824aa2593e1f2a4cf740343f" 
> 
> s2 := "dee2bd5dde763885944bc9d65419"
> 
> s3 := s1 + s2 
> 
> s1s2Byte, _ := hex.DecodeString(s3)
> 
> randAutnBin := fmt.Sprintf("%b", s1s2Byte)
> 
> result:
> 
> [11101 101000 11101101 1100110 1010 1001010 10100010 1011001 10 1 
> 101010 1001100 0111 100 110100 11 1100 11100010 1001 
> 1011101 1100 1110110 111000 1101 10010100 1001011 11001001 11010110 
> 1010100 11001]
> 
> I would like to decode the binary result back the hexadecimal string to get 
> s1 and s2.
> 
> Any help?
> 
> Van
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com <>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/ad3a981b-cf22-47cd-9fe6-8db83a097b42n%40googlegroups.com
>  
> .
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com <>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1f66ccdc-6c58-48f0-9017-9614220f88d9n%40googlegroups.com
>  
> .
> 
> -- 
> You received this message because you are subscribed to a 

Re: [go-nuts] Detecting / Avoiding network bottleneck when sending concurrent HTTP requests to many hosts.

[Robert Engels]

Have you used netstat to check how many open connections you have? You probably 
need to forcibly close them so they don’t end up in time wait states. 

Also, you don’t say what N is?

Lastly you will get different errors for a dns or host error than a resources 
exhausted. If you configure the maximum sockets/files allowed to be open by the 
process you won’t DOS the machine. 

> On Mar 6, 2023, at 8:07 AM, nohehf  wrote:
> 
> Hi! I'm building a tool that needs to concurrently send request to multiple 
> hosts (for fingerprinting purposes). 
> I'm using github.com/valyala/fasthttp for the client but it works the same 
> with net/http. 
> The main program basically starts n workers and those will take endpoints to 
> scan from a common channel.
> It's working fine but as I try to increment the number of workers at some 
> point i'll basically DOSing the machine it runs on which prevents some/all 
> requests to be sent, and even impact any other service running in the machine 
> (k8s cluster in fact).
> Any Idea on how I can detect this kind of network bottleneck within my 
> program ?
> I'd like to be able to maximise speed (number of workers) without destroying 
> the network. I can't really rely on the hosts responses to detect if the 
> network is flooded as a DNS not found or request timeout can happen 
> legitimately on some hosts (the program can scan non existent / very slow 
> hosts).
> Thank you!
> 
> PS: I'm pretty new to go and I'm a first time poster here, so please 
> understand my lack of knowledge if any :)
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/60b4a477-2731-4609-a927-bb33ad3d409bn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A7628E9E-2C44-4397-9001-E0994544CF4A%40ix.netcom.com.

Re: [go-nuts] go --> wasm: syscall.Mmap

[Robert Engels]

See https://github.com/WebAssembly/WASI/issues/304

> On Feb 17, 2023, at 2:44 PM, Robert Engels  wrote:
> 
> You must like need to use WASI - but it doesn’t support mmap :(
> 
>>> On Feb 17, 2023, at 2:37 PM, Ian Lance Taylor  wrote:
>>> 
>>> On Fri, Feb 17, 2023 at 9:11 AM Stan Srednyak  wrote:
>>> 
>>> thanks for your reply.
>>> The Mmap functionality is crucial for my app. I am dealing with large 
>>> binary trees that do not fit RAM. I implemented them on disk, and for this, 
>>> Mmap is crucial.
>>> 
>>> Now I have a client that wants to run the app from inside a virtual box, 
>>> and for security reason, they want .wasm code.
>> 
>> Sorry, I have no idea whether this is possible using wasm.  I suspect
>> that it is not but I am not an expert.
>> 
>> Ian
>> 
>> 
>>>> On Fri, Feb 17, 2023 at 11:14 AM Ian Lance Taylor  wrote:
>>>> 
>>>> On Fri, Feb 17, 2023, 6:47 AM Stan Srednyak  wrote:
>>>>> 
>>>>> I have a Go code that I am trying to compile to .wasm. The code uses 
>>>>> syscall.Mmap.
>>>>> 
>>>>> I am getting  undefined: syscall.Mmap
>>>>> 
>>>>> 
>>>>> What is this error and what do I do with it?
>>>> 
>>>> 
>>>> The syscall package provides system dependent functionality.  Wasm does 
>>>> not support memory mapping, so syscall.Mmap does not exist on Wasm.
>>>> 
>>>> In order to make any useful suggestions we would have to know why your Go 
>>>> code is calling Mmap.
>>>> 
>>>> Ian
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX-2SgB%3Dq9bPKSxugfKD4-yTNFPUmSsF8zA5mdfzOrYrw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9189930A-A820-460E-B9CD-36FA2DB7339D%40ix.netcom.com.

Re: [go-nuts] go --> wasm: syscall.Mmap

[Robert Engels]

You must like need to use WASI - but it doesn’t support mmap :(

> On Feb 17, 2023, at 2:37 PM, Ian Lance Taylor  wrote:
> 
> On Fri, Feb 17, 2023 at 9:11 AM Stan Srednyak  wrote:
>> 
>> thanks for your reply.
>> The Mmap functionality is crucial for my app. I am dealing with large binary 
>> trees that do not fit RAM. I implemented them on disk, and for this, Mmap is 
>> crucial.
>> 
>> Now I have a client that wants to run the app from inside a virtual box, and 
>> for security reason, they want .wasm code.
> 
> Sorry, I have no idea whether this is possible using wasm.  I suspect
> that it is not but I am not an expert.
> 
> Ian
> 
> 
>>> On Fri, Feb 17, 2023 at 11:14 AM Ian Lance Taylor  wrote:
>>> 
>>> On Fri, Feb 17, 2023, 6:47 AM Stan Srednyak  wrote:
 
 I have a Go code that I am trying to compile to .wasm. The code uses 
 syscall.Mmap.
 
 I am getting  undefined: syscall.Mmap
 
 
 What is this error and what do I do with it?
>>> 
>>> 
>>> The syscall package provides system dependent functionality.  Wasm does not 
>>> support memory mapping, so syscall.Mmap does not exist on Wasm.
>>> 
>>> In order to make any useful suggestions we would have to know why your Go 
>>> code is calling Mmap.
>>> 
>>> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX-2SgB%3Dq9bPKSxugfKD4-yTNFPUmSsF8zA5mdfzOrYrw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/581ADB02-F4A4-4B7F-A95A-5034C3604D4A%40ix.netcom.com.

Re: [go-nuts] Re: Upgradable RLock

[Robert Engels]

ConcurrentHashMap works that way in Java. Multiple write locks on portions of 
the table and CAS/atomic reading. 

> On Feb 13, 2023, at 1:45 PM, 'drc...@google.com' via golang-nuts 
>  wrote:
> 
> Could you use an applicative data structure? (e.g., a balanced binary tree 
> where you allocate a new spine for each insertion/deletion)
> That has log N overhead to read, log N storage allocated per write, but I 
> think if you CAS the writes, the reads can proceed with a lightweight barrier.
> 
> 
>> On Sunday, February 5, 2023 at 11:45:38 AM UTC-5 Ian Lance Taylor wrote:
>>> On Sun, Feb 5, 2023, 4:34 AM Juliusz Chroboczek  wrote:
>>> >> I took some time to put this to a test. The Go program here
>>> >> https://go.dev/play/p/378Zn_ZQNaz uses a VERY short holding of the
>>> >> lock - but a large % of runtime holding the lock.
>>> 
>>> > Thanks for the benchmark.  You're right: if you have hundreds of
>>> > goroutines doing nothing but acquiring a read lock, then an RWMutex
>>> > can be faster.  They key there is that there are always multiple
>>> > goroutines waiting for the lock.
>>> 
>>> Could you please explain that?  You previously implied that the
>>> required atomic operation is going to make the difference between the
>>> two kinds of mutex irrelevant, why does the argument not apply here?
>> 
>> 
>> If there are enough concurrent lockers to overwhelm the plain mutex spin 
>> lock, then the read-write mutex will work better.  My argument is that in 
>> real programs that is an unlikely case if the lock is held for a short 
>> period of time.
>> 
>> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3769f0c0-dc9b-417f-8be0-6113a4fe0090n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EFE47D4A-CF44-4BAB-B9DE-05258135C658%40ix.netcom.com.

[go-nuts] Re: Re-entrant locks?

[Robert Engels]

And you have to make sure that a ctx is not incorrectly shared… ugh. 

> On Feb 7, 2023, at 7:41 AM, Robert Engels  wrote:
> 
> I know this topic comes up a bit and I’ve done a bit more research. 
> 
> Is a contributing reason that Go does not have re-entrant locks is that Go 
> does not have any way to get Go routine identity? It seems you cannot write a 
> reentrant lock - at least not with the typical syntax - without specialized 
> runtime support. 
> 
> The only public way I can think of is something like
> 
> somelock.Lock(ctx)
> 
> But then you need to make every method accept a context.Context instance.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/D4211DBD-3147-41E0-B679-C0569DC2029B%40ix.netcom.com.

[go-nuts] Re-entrant locks?

[Robert Engels]

I know this topic comes up a bit and I’ve done a bit more research. 

Is a contributing reason that Go does not have re-entrant locks is that Go does 
not have any way to get Go routine identity? It seems you cannot write a 
reentrant lock - at least not with the typical syntax - without specialized 
runtime support. 

The only public way I can think of is something like

somelock.Lock(ctx)

But then you need to make every method accept a context.Context instance. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/69B98553-62B3-44BC-B3D9-B0001214139C%40ix.netcom.com.

Re: [go-nuts] Upgradable RLock

[robert engels]

That is similar to sync.Map works, but it involves much more complex code.

More importantly though, if multiple entries need to be keep in sync that 
technique doesn’t work - at least not directly/easily. This is a common need 
with associated caches.

Even copy on write isn’t always suitable. Assume you have a map (cache) that is 
1GB in size. It is mainly read. But you need to update an entry every once in a 
while.

With copy on write, the “create a new value” - needs to create a new map and 
copy over the existing map - very expensive. Then atomically replace the 
reference.

With multiple writers this can be even more expensive, since you need a 
secondary lock to avoid each writer attempting to make an expensive copy then 
failing the CAS. (no more expensive than the write lock in RWMutex).


> On Feb 4, 2023, at 6:14 PM, Ian Lance Taylor  <mailto:i...@golang.org>> wrote:
> 
> On Sat, Feb 4, 2023 at 3:24 PM Robert Engels  <mailto:reng...@ix.netcom.com>> wrote:
>> 
>> That only works if what it is pointing to is cheap to copy. If it is a large 
>> multi-layer structure a RW lock is usually more efficient.
> 
> No significant copying is required, you just get a pointer to the
> value.  Then you have some way to determine whether it is up to date.
> If not, you create a new value and store a pointer to it back in the
> atomic.Pointer.
> 
> Ian
> 
> 
>>> On Feb 4, 2023, at 5:19 PM, Ian Lance Taylor >> <mailto:i...@golang.org>> wrote:
>>> 
>>> On Sat, Feb 4, 2023 at 3:11 PM Robert Engels >> <mailto:reng...@ix.netcom.com>> wrote:
>>>> 
>>>> I think with server processes - with possibly 100k+ connections - the 
>>>> contention on a “read mainly” cache is more than you think. This test only 
>>>> uses 500 readers with little work to simulate the 100k case.
>>> 
>>> Not to get too far into the weeds, but if I were expecting that kind
>>> of load I would use an atomic.Pointer anyhow, rather than any sort of
>>> mutex.
>>> 
>>> Ian
>>> 
>>>>>> On Feb 4, 2023, at 4:59 PM, Ian Lance Taylor >>>>> <mailto:i...@golang.org>> wrote:
>>>>> 
>>>>> On Sat, Feb 4, 2023 at 8:49 AM robert engels >>>> <mailto:reng...@ix.netcom.com>> wrote:
>>>>>> 
>>>>>> I took some time to put this to a test. The Go program here 
>>>>>> https://go.dev/play/p/378Zn_ZQNaz <https://go.dev/play/p/378Zn_ZQNaz> 
>>>>>> uses a VERY short holding of the lock - but a large % of runtime holding 
>>>>>> the lock.
>>>>>> 
>>>>>> (You can’t run it on the Playground because of the length of time). You 
>>>>>> can comment/uncomment the lines 28-31 to test the different mutexes,
>>>>>> 
>>>>>> It simulates a common system scenario (most web services) - lots of 
>>>>>> readers of the cache, but the cache is updated infrequently.
>>>>>> 
>>>>>> On my machine the RWMutex is > 50% faster - taking 22 seconds vs 47 
>>>>>> seconds using a simple Mutex.
>>>>>> 
>>>>>> It is easy to understand why - you get no parallelization of the readers 
>>>>>> when using a simple Mutex.
>>>>> 
>>>>> Thanks for the benchmark.  You're right: if you have hundreds of
>>>>> goroutines doing nothing but acquiring a read lock, then an RWMutex
>>>>> can be faster.  They key there is that there are always multiple
>>>>> goroutines waiting for the lock.
>>>>> 
>>>>> I still stand by my statement for more common use cases.
>>>>> 
>>>>> Ian
>>>>> 
>>>>> 
>>>>>> On Jan 30, 2023, at 8:29 PM, Ian Lance Taylor >>>>> <mailto:i...@golang.org>> wrote:
>>>>>> 
>>>>>> On Mon, Jan 30, 2023 at 4:42 PM Robert Engels >>>>> <mailto:reng...@ix.netcom.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> Yes but only for a single reader - any concurrent reader is going to 
>>>>>> park/deschedule.
>>>>>> 
>>>>>> 
>>>>>> If we are talking specifically about Go, then it's more complex than
>>>>>> that.  In particular, the code will spin briefly trying to acquire the
>>>>>> mutex, before queuing.
>>>>>> 
>>>>>> There’s a reas

Re: [go-nuts] Upgradable RLock

[Robert Engels]

That only works if what it is pointing to is cheap to copy. If it is a large 
multi-layer structure a RW lock is usually more efficient. 


> On Feb 4, 2023, at 5:19 PM, Ian Lance Taylor  wrote:
> 
> On Sat, Feb 4, 2023 at 3:11 PM Robert Engels  wrote:
>> 
>> I think with server processes - with possibly 100k+ connections - the 
>> contention on a “read mainly” cache is more than you think. This test only 
>> uses 500 readers with little work to simulate the 100k case.
> 
> Not to get too far into the weeds, but if I were expecting that kind
> of load I would use an atomic.Pointer anyhow, rather than any sort of
> mutex.
> 
> Ian
> 
>>>> On Feb 4, 2023, at 4:59 PM, Ian Lance Taylor  wrote:
>>> 
>>> On Sat, Feb 4, 2023 at 8:49 AM robert engels  wrote:
>>>> 
>>>> I took some time to put this to a test. The Go program here 
>>>> https://go.dev/play/p/378Zn_ZQNaz uses a VERY short holding of the lock - 
>>>> but a large % of runtime holding the lock.
>>>> 
>>>> (You can’t run it on the Playground because of the length of time). You 
>>>> can comment/uncomment the lines 28-31 to test the different mutexes,
>>>> 
>>>> It simulates a common system scenario (most web services) - lots of 
>>>> readers of the cache, but the cache is updated infrequently.
>>>> 
>>>> On my machine the RWMutex is > 50% faster - taking 22 seconds vs 47 
>>>> seconds using a simple Mutex.
>>>> 
>>>> It is easy to understand why - you get no parallelization of the readers 
>>>> when using a simple Mutex.
>>> 
>>> Thanks for the benchmark.  You're right: if you have hundreds of
>>> goroutines doing nothing but acquiring a read lock, then an RWMutex
>>> can be faster.  They key there is that there are always multiple
>>> goroutines waiting for the lock.
>>> 
>>> I still stand by my statement for more common use cases.
>>> 
>>> Ian
>>> 
>>> 
>>>> On Jan 30, 2023, at 8:29 PM, Ian Lance Taylor  wrote:
>>>> 
>>>> On Mon, Jan 30, 2023 at 4:42 PM Robert Engels  
>>>> wrote:
>>>> 
>>>> 
>>>> Yes but only for a single reader - any concurrent reader is going to 
>>>> park/deschedule.
>>>> 
>>>> 
>>>> If we are talking specifically about Go, then it's more complex than
>>>> that.  In particular, the code will spin briefly trying to acquire the
>>>> mutex, before queuing.
>>>> 
>>>> There’s a reason RW locks exist - and I think it is pretty common - but 
>>>> agree to disagree :)
>>>> 
>>>> 
>>>> Sure: read-write locks are fine and appropriate when the program holds
>>>> the read lock for a reasonably lengthy time.  As I said, my analysis
>>>> only applies when code holds the read lock briefly, as is often the
>>>> case for a cache.
>>>> 
>>>> Ian
>>>> 
>>>> 
>>>> On Jan 30, 2023, at 6:23 PM, Ian Lance Taylor  wrote:
>>>> 
>>>> On Mon, Jan 30, 2023 at 1:00 PM Robert Engels  
>>>> wrote:
>>>> 
>>>> 
>>>> Pure readers do not need any mutex on the fast path. It is an atomic CAS - 
>>>> which is faster than a mutex as it allows concurrent readers. On the slow 
>>>> path - fairness with a waiting or active writer - it degenerates in 
>>>> performance to a simple mutex.
>>>> 
>>>> The issue with a mutex is that you need to acquire it whether reading or 
>>>> writing - this is slow…. (at least compared to an atomic cas)
>>>> 
>>>> 
>>>> The fast path of a mutex is also an atomic CAS.
>>>> 
>>>> Ian
>>>> 
>>>> On Jan 30, 2023, at 2:24 PM, Ian Lance Taylor  wrote:
>>>> 
>>>> 
>>>> On Mon, Jan 30, 2023 at 11:26 AM Robert Engels  
>>>> wrote:
>>>> 
>>>> 
>>>> I don’t think that is true. A RW lock is always better when the reader 
>>>> activity is far greater than the writer - simply because in a good 
>>>> implementation the read lock can be acquired without blocking/scheduling 
>>>> activity.
>>>> 
>>>> 
>>>> The best read lock implementation is not going to be better than the
>>>> best plain mutex implementation.  And with current technology any
>>>> impleme

Re: [go-nuts] Upgradable RLock

[Robert Engels]

I think with server processes - with possibly 100k+ connections - the 
contention on a “read mainly” cache is more than you think. This test only uses 
500 readers with little work to simulate the 100k case. 

> On Feb 4, 2023, at 4:59 PM, Ian Lance Taylor  wrote:
> 
> On Sat, Feb 4, 2023 at 8:49 AM robert engels  wrote:
>> 
>> I took some time to put this to a test. The Go program here 
>> https://go.dev/play/p/378Zn_ZQNaz uses a VERY short holding of the lock - 
>> but a large % of runtime holding the lock.
>> 
>> (You can’t run it on the Playground because of the length of time). You can 
>> comment/uncomment the lines 28-31 to test the different mutexes,
>> 
>> It simulates a common system scenario (most web services) - lots of readers 
>> of the cache, but the cache is updated infrequently.
>> 
>> On my machine the RWMutex is > 50% faster - taking 22 seconds vs 47 seconds 
>> using a simple Mutex.
>> 
>> It is easy to understand why - you get no parallelization of the readers 
>> when using a simple Mutex.
> 
> Thanks for the benchmark.  You're right: if you have hundreds of
> goroutines doing nothing but acquiring a read lock, then an RWMutex
> can be faster.  They key there is that there are always multiple
> goroutines waiting for the lock.
> 
> I still stand by my statement for more common use cases.
> 
> Ian
> 
> 
>> On Jan 30, 2023, at 8:29 PM, Ian Lance Taylor  wrote:
>> 
>> On Mon, Jan 30, 2023 at 4:42 PM Robert Engels  wrote:
>> 
>> 
>> Yes but only for a single reader - any concurrent reader is going to 
>> park/deschedule.
>> 
>> 
>> If we are talking specifically about Go, then it's more complex than
>> that.  In particular, the code will spin briefly trying to acquire the
>> mutex, before queuing.
>> 
>> There’s a reason RW locks exist - and I think it is pretty common - but 
>> agree to disagree :)
>> 
>> 
>> Sure: read-write locks are fine and appropriate when the program holds
>> the read lock for a reasonably lengthy time.  As I said, my analysis
>> only applies when code holds the read lock briefly, as is often the
>> case for a cache.
>> 
>> Ian
>> 
>> 
>> On Jan 30, 2023, at 6:23 PM, Ian Lance Taylor  wrote:
>> 
>> On Mon, Jan 30, 2023 at 1:00 PM Robert Engels  wrote:
>> 
>> 
>> Pure readers do not need any mutex on the fast path. It is an atomic CAS - 
>> which is faster than a mutex as it allows concurrent readers. On the slow 
>> path - fairness with a waiting or active writer - it degenerates in 
>> performance to a simple mutex.
>> 
>> The issue with a mutex is that you need to acquire it whether reading or 
>> writing - this is slow…. (at least compared to an atomic cas)
>> 
>> 
>> The fast path of a mutex is also an atomic CAS.
>> 
>> Ian
>> 
>> On Jan 30, 2023, at 2:24 PM, Ian Lance Taylor  wrote:
>> 
>> 
>> On Mon, Jan 30, 2023 at 11:26 AM Robert Engels  
>> wrote:
>> 
>> 
>> I don’t think that is true. A RW lock is always better when the reader 
>> activity is far greater than the writer - simply because in a good 
>> implementation the read lock can be acquired without blocking/scheduling 
>> activity.
>> 
>> 
>> The best read lock implementation is not going to be better than the
>> best plain mutex implementation.  And with current technology any
>> implementation is going to require atomic memory operations which
>> require coordinating cache lines between CPUs.  If your reader
>> activity is so large that you get significant contention on a plain
>> mutex (recalling that we are assuming the case where the operations
>> under the read lock are quick) then you are also going to get
>> significant contention on a read lock.  The effect is that the read
>> lock isn't going to be faster anyhow in practice, and your program
>> should probably be using a different approach.
>> 
>> Ian
>> 
>> On Jan 30, 2023, at 12:49 PM, Ian Lance Taylor  wrote:
>> 
>> 
>> On Sun, Jan 29, 2023 at 6:34 PM Diego Augusto Molina
>>  wrote:
>> 
>> 
>> From times to times I write a scraper or some other tool that would 
>> authenticate to a service and then use the auth result to do stuff 
>> concurrently. But when auth expires, I need to synchronize all my goroutines 
>> and have a single one do the re-auth process, check the status, etc. and 
>> then arrange for all goroutines to go back to work using the new auth result.
>> 
>> To generalize the proble

Re: [go-nuts] Upgradable RLock

[Robert Engels]

Agreed. Copy-on-write is also very good if the shared structure is small. 

> On Feb 4, 2023, at 1:00 PM, Kurtis Rader  wrote:
> 
> 
> FWIW, Using an RCU (https://en.wikipedia.org/wiki/Read-copy-update) algorithm 
> is often the best choice for caches where reads happen several orders of 
> magnitude more often than writes. RCU usually avoids the need for any mutex 
> by readers. I was working at Sequent Computer Systems when this was 
> implemented for the network ARP and routing tables and was friends with the 
> engineers who designed and patented RCU. I remember reading their patent and 
> looking at the kernel implementation and thinking to myself how brilliant the 
> concept was.
> 
>> On Sat, Feb 4, 2023 at 8:49 AM robert engels  wrote:
>> I took some time to put this to a test. The Go program here 
>> https://go.dev/play/p/378Zn_ZQNaz uses a VERY short holding of the lock - 
>> but a large % of runtime holding the lock.
>> 
>> (You can’t run it on the Playground because of the length of time). You can 
>> comment/uncomment the lines 28-31 to test the different mutexes,
>> 
>> It simulates a common system scenario (most web services) - lots of readers 
>> of the cache, but the cache is updated infrequently.
>> 
>> On my machine the RWMutex is > 50% faster - taking 22 seconds vs 47 seconds 
>> using a simple Mutex.
>> 
>> It is easy to understand why - you get no parallelization of the readers 
>> when using a simple Mutex. 
>> 
>>> On Jan 30, 2023, at 8:29 PM, Ian Lance Taylor  wrote:
>>> 
>>>> On Mon, Jan 30, 2023 at 4:42 PM Robert Engels  
>>>> wrote:
>>>> 
>>>> Yes but only for a single reader - any concurrent reader is going to 
>>>> park/deschedule.
>>> 
>>> If we are talking specifically about Go, then it's more complex than
>>> that.  In particular, the code will spin briefly trying to acquire the
>>> mutex, before queuing.
>>> 
>>>> There’s a reason RW locks exist - and I think it is pretty common - but 
>>>> agree to disagree :)
>>> 
>>> Sure: read-write locks are fine and appropriate when the program holds
>>> the read lock for a reasonably lengthy time.  As I said, my analysis
>>> only applies when code holds the read lock briefly, as is often the
>>> case for a cache.
>>> 
>>> Ian
>>> 
>>> 
>>>>> On Jan 30, 2023, at 6:23 PM, Ian Lance Taylor  wrote:
>>>>> 
>>>>> On Mon, Jan 30, 2023 at 1:00 PM Robert Engels  
>>>>> wrote:
>>>>>> 
>>>>>> Pure readers do not need any mutex on the fast path. It is an atomic CAS 
>>>>>> - which is faster than a mutex as it allows concurrent readers. On the 
>>>>>> slow path - fairness with a waiting or active writer - it degenerates in 
>>>>>> performance to a simple mutex.
>>>>>> 
>>>>>> The issue with a mutex is that you need to acquire it whether reading or 
>>>>>> writing - this is slow…. (at least compared to an atomic cas)
>>>>> 
>>>>> The fast path of a mutex is also an atomic CAS.
>>>>> 
>>>>> Ian
>>>>> 
>>>>>>>> On Jan 30, 2023, at 2:24 PM, Ian Lance Taylor  wrote:
>>>>>>> 
>>>>>>> On Mon, Jan 30, 2023 at 11:26 AM Robert Engels  
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> I don’t think that is true. A RW lock is always better when the reader 
>>>>>>>> activity is far greater than the writer - simply because in a good 
>>>>>>>> implementation the read lock can be acquired without 
>>>>>>>> blocking/scheduling activity.
>>>>>>> 
>>>>>>> The best read lock implementation is not going to be better than the
>>>>>>> best plain mutex implementation.  And with current technology any
>>>>>>> implementation is going to require atomic memory operations which
>>>>>>> require coordinating cache lines between CPUs.  If your reader
>>>>>>> activity is so large that you get significant contention on a plain
>>>>>>> mutex (recalling that we are assuming the case where the operations
>>>>>>> under the read lock are quick) then you are also going to get
>>>>>>> significant contention on a read lock.  The effect is that the read
>>>&

Re: [go-nuts] Upgradable RLock

[robert engels]

I took some time to put this to a test. The Go program here 
https://go.dev/play/p/378Zn_ZQNaz <https://go.dev/play/p/rNJNbek4ufm> uses a 
VERY short holding of the lock - but a large % of runtime holding the lock.

(You can’t run it on the Playground because of the length of time). You can 
comment/uncomment the lines 28-31 to test the different mutexes,

It simulates a common system scenario (most web services) - lots of readers of 
the cache, but the cache is updated infrequently.

On my machine the RWMutex is > 50% faster - taking 22 seconds vs 47 seconds 
using a simple Mutex.

It is easy to understand why - you get no parallelization of the readers when 
using a simple Mutex. 

> On Jan 30, 2023, at 8:29 PM, Ian Lance Taylor  wrote:
> 
> On Mon, Jan 30, 2023 at 4:42 PM Robert Engels  wrote:
>> 
>> Yes but only for a single reader - any concurrent reader is going to 
>> park/deschedule.
> 
> If we are talking specifically about Go, then it's more complex than
> that.  In particular, the code will spin briefly trying to acquire the
> mutex, before queuing.
> 
>> There’s a reason RW locks exist - and I think it is pretty common - but 
>> agree to disagree :)
> 
> Sure: read-write locks are fine and appropriate when the program holds
> the read lock for a reasonably lengthy time.  As I said, my analysis
> only applies when code holds the read lock briefly, as is often the
> case for a cache.
> 
> Ian
> 
> 
>>> On Jan 30, 2023, at 6:23 PM, Ian Lance Taylor  wrote:
>>> 
>>> On Mon, Jan 30, 2023 at 1:00 PM Robert Engels  
>>> wrote:
>>>> 
>>>> Pure readers do not need any mutex on the fast path. It is an atomic CAS - 
>>>> which is faster than a mutex as it allows concurrent readers. On the slow 
>>>> path - fairness with a waiting or active writer - it degenerates in 
>>>> performance to a simple mutex.
>>>> 
>>>> The issue with a mutex is that you need to acquire it whether reading or 
>>>> writing - this is slow…. (at least compared to an atomic cas)
>>> 
>>> The fast path of a mutex is also an atomic CAS.
>>> 
>>> Ian
>>> 
>>>>>> On Jan 30, 2023, at 2:24 PM, Ian Lance Taylor  wrote:
>>>>> 
>>>>> On Mon, Jan 30, 2023 at 11:26 AM Robert Engels  
>>>>> wrote:
>>>>>> 
>>>>>> I don’t think that is true. A RW lock is always better when the reader 
>>>>>> activity is far greater than the writer - simply because in a good 
>>>>>> implementation the read lock can be acquired without blocking/scheduling 
>>>>>> activity.
>>>>> 
>>>>> The best read lock implementation is not going to be better than the
>>>>> best plain mutex implementation.  And with current technology any
>>>>> implementation is going to require atomic memory operations which
>>>>> require coordinating cache lines between CPUs.  If your reader
>>>>> activity is so large that you get significant contention on a plain
>>>>> mutex (recalling that we are assuming the case where the operations
>>>>> under the read lock are quick) then you are also going to get
>>>>> significant contention on a read lock.  The effect is that the read
>>>>> lock isn't going to be faster anyhow in practice, and your program
>>>>> should probably be using a different approach.
>>>>> 
>>>>> Ian
>>>>> 
>>>>>>>> On Jan 30, 2023, at 12:49 PM, Ian Lance Taylor  wrote:
>>>>>>> 
>>>>>>> On Sun, Jan 29, 2023 at 6:34 PM Diego Augusto Molina
>>>>>>>  wrote:
>>>>>>>> 
>>>>>>>> From times to times I write a scraper or some other tool that would 
>>>>>>>> authenticate to a service and then use the auth result to do stuff 
>>>>>>>> concurrently. But when auth expires, I need to synchronize all my 
>>>>>>>> goroutines and have a single one do the re-auth process, check the 
>>>>>>>> status, etc. and then arrange for all goroutines to go back to work 
>>>>>>>> using the new auth result.
>>>>>>>> 
>>>>>>>> To generalize the problem: multiple goroutines read a cached value 
>>>>>>>> that expires at some point. When it does, they all should block and 
>>>>>>>> some I/O operation has to be performed by a single goroutine to

Re: [go-nuts] please continue High Sierra mac OSX support

[Robert Engels]

That’s a reasonable position. Yea, I’ve haven’t  been too happy with Apple 
either in terms of newer OSes not working on older hardware - usually for no 
reason other than to drive hardware upgrades. 

> On Feb 3, 2023, at 12:33 PM, Ian Lance Taylor  wrote:
> 
> On Fri, Feb 3, 2023 at 9:40 AM robert engels  wrote:
>> 
>> I’d like to understand this a bit better as well. I currently develop an OSX 
>> app using Xcode/Obj-C and it runs all the way back to 10.9 (I recently 
>> raised the requirement from 10.7).
>> 
>> Is the restriction only that the Go tool chain needs 10.15, but the produced 
>> binaries will work on an earlier version? Because that seems reasonable.
>> 
>> Or do I have to use an old version of Go to produce binaries that run on an 
>> old version of OSX? This seems highly restrictive and harkens to the days of 
>> M$ and the continual upgrade cycles that churned through resources.
> 
> What dropping support means is that we no longer fix bugs that only
> occur on unsupported releases, and we no longer run our tests on old
> releases.  That includes both running the Go tools, and running
> programs produced by the Go tools.
> 
> We don't go out of our way to cause Go to break on unsupported
> releases.  That said, there are occasionally cases where we change Go
> to require APIs that are only available on newer releases.  That has
> happened on macOS in the sense that we no longer support 386 or arm at
> all; we only support amd64 and arm64.  I don't know offhand if there
> have been other such cases for macOS (there have been for Linux: for
> example, we now require the accept4 system call to be supported
> (except on arm)).
> 
> Our position is that it is actually Apple that is driving the upgrade
> cycle you mention, not us.  Apple chose to drop support for High
> Sierra back in 2020, so anybody running a High Sierra system connected
> to the Internet is at risk.  Should we invest our limited resources on
> supporting releases that even Apple has declined to support?  If Apple
> continued to support High Sierra, so would we.
> 
> Ian
> 
> 
>>>> On Feb 3, 2023, at 11:34 AM, Ian Lance Taylor  wrote:
>>> 
>>> On Fri, Feb 3, 2023 at 8:34 AM Jason E. Aten  wrote:
>>>> 
>>>> The Go 1.20 release notes say:
>>>> 
>>>>> Go 1.20 is the last release that will run on macOS 10.13 High Sierra or 
>>>>> 10.14 Mojave. Go 1.21 will require macOS 10.15 Catalina or later.
>>>> 
>>>> This is sad to hear, since High Sierra is the preferred (most stable) 
>>>> available mac operating system.
>>>> 
>>>> Please consider continuing to support High Sierra in future Go versions.
>>> 
>>> Our general guideline is that we stop supporting an operating system
>>> when that operating system is itself no longer supported.  According
>>> to Wikipedia, Apple stopped supporting High Sierra in 2020, so
>>> presumably people should be moving off it.  It's also going to be
>>> steadily harder for us to test it.  What's the argument for us
>>> continuing to support it?
>>> 
>>> Ian
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXUQLknkjw8Hk345bO6EazuEZcjKwvAw1AEEiCkz5OUwQ%40mail.gmail.com.
>> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/133B404A-283E-4060-956D-EFD3806556A5%40ix.netcom.com.

Re: [go-nuts] please continue High Sierra mac OSX support

[robert engels]

I’d like to understand this a bit better as well. I currently develop an OSX 
app using Xcode/Obj-C and it runs all the way back to 10.9 (I recently raised 
the requirement from 10.7).

Is the restriction only that the Go tool chain needs 10.15, but the produced 
binaries will work on an earlier version? Because that seems reasonable.

Or do I have to use an old version of Go to produce binaries that run on an old 
version of OSX? This seems highly restrictive and harkens to the days of M$ and 
the continual upgrade cycles that churned through resources.

> On Feb 3, 2023, at 11:34 AM, Ian Lance Taylor  wrote:
> 
> On Fri, Feb 3, 2023 at 8:34 AM Jason E. Aten  wrote:
>> 
>> The Go 1.20 release notes say:
>> 
>>> Go 1.20 is the last release that will run on macOS 10.13 High Sierra or 
>>> 10.14 Mojave. Go 1.21 will require macOS 10.15 Catalina or later.
>> 
>> This is sad to hear, since High Sierra is the preferred (most stable) 
>> available mac operating system.
>> 
>> Please consider continuing to support High Sierra in future Go versions.
> 
> Our general guideline is that we stop supporting an operating system
> when that operating system is itself no longer supported.  According
> to Wikipedia, Apple stopped supporting High Sierra in 2020, so
> presumably people should be moving off it.  It's also going to be
> steadily harder for us to test it.  What's the argument for us
> continuing to support it?
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXUQLknkjw8Hk345bO6EazuEZcjKwvAw1AEEiCkz5OUwQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/071BB6B1-7B1F-431D-A28D-D05FD3EDC1FA%40ix.netcom.com.

Re: [go-nuts] Upgradable RLock

[Robert Engels]

Yes but only for a single reader - any concurrent reader is going to 
park/deschedule. 

There’s a reason RW locks exist - and I think it is pretty common - but agree 
to disagree :)

> On Jan 30, 2023, at 6:23 PM, Ian Lance Taylor  wrote:
> 
> On Mon, Jan 30, 2023 at 1:00 PM Robert Engels  wrote:
>> 
>> Pure readers do not need any mutex on the fast path. It is an atomic CAS - 
>> which is faster than a mutex as it allows concurrent readers. On the slow 
>> path - fairness with a waiting or active writer - it degenerates in 
>> performance to a simple mutex.
>> 
>> The issue with a mutex is that you need to acquire it whether reading or 
>> writing - this is slow…. (at least compared to an atomic cas)
> 
> The fast path of a mutex is also an atomic CAS.
> 
> Ian
> 
>>>> On Jan 30, 2023, at 2:24 PM, Ian Lance Taylor  wrote:
>>> 
>>> On Mon, Jan 30, 2023 at 11:26 AM Robert Engels  
>>> wrote:
>>>> 
>>>> I don’t think that is true. A RW lock is always better when the reader 
>>>> activity is far greater than the writer - simply because in a good 
>>>> implementation the read lock can be acquired without blocking/scheduling 
>>>> activity.
>>> 
>>> The best read lock implementation is not going to be better than the
>>> best plain mutex implementation.  And with current technology any
>>> implementation is going to require atomic memory operations which
>>> require coordinating cache lines between CPUs.  If your reader
>>> activity is so large that you get significant contention on a plain
>>> mutex (recalling that we are assuming the case where the operations
>>> under the read lock are quick) then you are also going to get
>>> significant contention on a read lock.  The effect is that the read
>>> lock isn't going to be faster anyhow in practice, and your program
>>> should probably be using a different approach.
>>> 
>>> Ian
>>> 
>>>>>> On Jan 30, 2023, at 12:49 PM, Ian Lance Taylor  wrote:
>>>>> 
>>>>> On Sun, Jan 29, 2023 at 6:34 PM Diego Augusto Molina
>>>>>  wrote:
>>>>>> 
>>>>>> From times to times I write a scraper or some other tool that would 
>>>>>> authenticate to a service and then use the auth result to do stuff 
>>>>>> concurrently. But when auth expires, I need to synchronize all my 
>>>>>> goroutines and have a single one do the re-auth process, check the 
>>>>>> status, etc. and then arrange for all goroutines to go back to work 
>>>>>> using the new auth result.
>>>>>> 
>>>>>> To generalize the problem: multiple goroutines read a cached value that 
>>>>>> expires at some point. When it does, they all should block and some I/O 
>>>>>> operation has to be performed by a single goroutine to renew the cached 
>>>>>> value, then unblock all other goroutines and have them use the new value.
>>>>>> 
>>>>>> I solved this in the past in a number of ways: having a single goroutine 
>>>>>> that handles the cache by asking it for the value through a channel, 
>>>>>> using sync.Cond (which btw every time I decide to use I need to 
>>>>>> carefully re-read its docs and do lots of tests because I never get it 
>>>>>> right at first). But what I came to do lately is to implement an 
>>>>>> upgradable lock and have every goroutine do:
>>>>> 
>>>>> 
>>>>> We have historically rejected this kind of adjustable lock.  There is
>>>>> some previous discussion at https://go.dev/issue/4026,
>>>>> https://go.dev/issue/23513, https://go.dev/issue/38891,
>>>>> https://go.dev/issue/44049.
>>>>> 
>>>>> For a cache where checking that the cached value is valid (not stale)
>>>>> and fetching the cached value is quick, then in general you will be
>>>>> better off using a plain Mutex rather than RWMutex.  RWMutex is more
>>>>> complicated and therefore slower.  It's only useful to use an RWMutex
>>>>> when the read case is both contested and relatively slow.  If the read
>>>>> case is fast then the simpler Mutex will tend to be faster.  And then
>>>>> you don't have to worry about upgrading the lock.
>>>>> 
>>>>> Ian
>>>>> 
>>>>> --
>>>>> You recei

Re: [go-nuts] Upgradable RLock

[Robert Engels]

A quick search found this https://yizhang82.dev/lock-free-rw-lock which 
describes the algo pretty well. 

> On Jan 30, 2023, at 3:00 PM, Robert Engels  wrote:
> 
> Pure readers do not need any mutex on the fast path. It is an atomic CAS - 
> which is faster than a mutex as it allows concurrent readers. On the slow 
> path - fairness with a waiting or active writer - it degenerates in 
> performance to a simple mutex. 
> 
> The issue with a mutex is that you need to acquire it whether reading or 
> writing - this is slow…. (at least compared to an atomic cas)
> 
>>> On Jan 30, 2023, at 2:24 PM, Ian Lance Taylor  wrote:
>>> 
>>> On Mon, Jan 30, 2023 at 11:26 AM Robert Engels  
>>> wrote:
>>> 
>>> I don’t think that is true. A RW lock is always better when the reader 
>>> activity is far greater than the writer - simply because in a good 
>>> implementation the read lock can be acquired without blocking/scheduling 
>>> activity.
>> 
>> The best read lock implementation is not going to be better than the
>> best plain mutex implementation.  And with current technology any
>> implementation is going to require atomic memory operations which
>> require coordinating cache lines between CPUs.  If your reader
>> activity is so large that you get significant contention on a plain
>> mutex (recalling that we are assuming the case where the operations
>> under the read lock are quick) then you are also going to get
>> significant contention on a read lock.  The effect is that the read
>> lock isn't going to be faster anyhow in practice, and your program
>> should probably be using a different approach.
>> 
>> Ian
>> 
>>>>> On Jan 30, 2023, at 12:49 PM, Ian Lance Taylor  wrote:
>>>> 
>>>> On Sun, Jan 29, 2023 at 6:34 PM Diego Augusto Molina
>>>>  wrote:
>>>>> 
>>>>> From times to times I write a scraper or some other tool that would 
>>>>> authenticate to a service and then use the auth result to do stuff 
>>>>> concurrently. But when auth expires, I need to synchronize all my 
>>>>> goroutines and have a single one do the re-auth process, check the 
>>>>> status, etc. and then arrange for all goroutines to go back to work using 
>>>>> the new auth result.
>>>>> 
>>>>> To generalize the problem: multiple goroutines read a cached value that 
>>>>> expires at some point. When it does, they all should block and some I/O 
>>>>> operation has to be performed by a single goroutine to renew the cached 
>>>>> value, then unblock all other goroutines and have them use the new value.
>>>>> 
>>>>> I solved this in the past in a number of ways: having a single goroutine 
>>>>> that handles the cache by asking it for the value through a channel, 
>>>>> using sync.Cond (which btw every time I decide to use I need to carefully 
>>>>> re-read its docs and do lots of tests because I never get it right at 
>>>>> first). But what I came to do lately is to implement an upgradable lock 
>>>>> and have every goroutine do:
>>>> 
>>>> 
>>>> We have historically rejected this kind of adjustable lock.  There is
>>>> some previous discussion at https://go.dev/issue/4026,
>>>> https://go.dev/issue/23513, https://go.dev/issue/38891,
>>>> https://go.dev/issue/44049.
>>>> 
>>>> For a cache where checking that the cached value is valid (not stale)
>>>> and fetching the cached value is quick, then in general you will be
>>>> better off using a plain Mutex rather than RWMutex.  RWMutex is more
>>>> complicated and therefore slower.  It's only useful to use an RWMutex
>>>> when the read case is both contested and relatively slow.  If the read
>>>> case is fast then the simpler Mutex will tend to be faster.  And then
>>>> you don't have to worry about upgrading the lock.
>>>> 
>>>> Ian
>>>> 
>>>> --
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXNVFkc5H-L6K4Mt81gB6u91Ja07hob%3DS8Qwgy2buiQjQ%40mail.gmail.com.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWJ%2BLPOoTk9H7bxAj8_dLsuhgOpy_bZZrGW%3D%2Bz6N%3DrX-w%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/DF457770-FC72-4AA4-AA7C-84B3900FFD95%40ix.netcom.com.

Re: [go-nuts] Upgradable RLock

[Robert Engels]

Pure readers do not need any mutex on the fast path. It is an atomic CAS - 
which is faster than a mutex as it allows concurrent readers. On the slow path 
- fairness with a waiting or active writer - it degenerates in performance to a 
simple mutex. 

The issue with a mutex is that you need to acquire it whether reading or 
writing - this is slow…. (at least compared to an atomic cas)

> On Jan 30, 2023, at 2:24 PM, Ian Lance Taylor  wrote:
> 
> On Mon, Jan 30, 2023 at 11:26 AM Robert Engels  wrote:
>> 
>> I don’t think that is true. A RW lock is always better when the reader 
>> activity is far greater than the writer - simply because in a good 
>> implementation the read lock can be acquired without blocking/scheduling 
>> activity.
> 
> The best read lock implementation is not going to be better than the
> best plain mutex implementation.  And with current technology any
> implementation is going to require atomic memory operations which
> require coordinating cache lines between CPUs.  If your reader
> activity is so large that you get significant contention on a plain
> mutex (recalling that we are assuming the case where the operations
> under the read lock are quick) then you are also going to get
> significant contention on a read lock.  The effect is that the read
> lock isn't going to be faster anyhow in practice, and your program
> should probably be using a different approach.
> 
> Ian
> 
>>>> On Jan 30, 2023, at 12:49 PM, Ian Lance Taylor  wrote:
>>> 
>>> On Sun, Jan 29, 2023 at 6:34 PM Diego Augusto Molina
>>>  wrote:
>>>> 
>>>> From times to times I write a scraper or some other tool that would 
>>>> authenticate to a service and then use the auth result to do stuff 
>>>> concurrently. But when auth expires, I need to synchronize all my 
>>>> goroutines and have a single one do the re-auth process, check the status, 
>>>> etc. and then arrange for all goroutines to go back to work using the new 
>>>> auth result.
>>>> 
>>>> To generalize the problem: multiple goroutines read a cached value that 
>>>> expires at some point. When it does, they all should block and some I/O 
>>>> operation has to be performed by a single goroutine to renew the cached 
>>>> value, then unblock all other goroutines and have them use the new value.
>>>> 
>>>> I solved this in the past in a number of ways: having a single goroutine 
>>>> that handles the cache by asking it for the value through a channel, using 
>>>> sync.Cond (which btw every time I decide to use I need to carefully 
>>>> re-read its docs and do lots of tests because I never get it right at 
>>>> first). But what I came to do lately is to implement an upgradable lock 
>>>> and have every goroutine do:
>>> 
>>> 
>>> We have historically rejected this kind of adjustable lock.  There is
>>> some previous discussion at https://go.dev/issue/4026,
>>> https://go.dev/issue/23513, https://go.dev/issue/38891,
>>> https://go.dev/issue/44049.
>>> 
>>> For a cache where checking that the cached value is valid (not stale)
>>> and fetching the cached value is quick, then in general you will be
>>> better off using a plain Mutex rather than RWMutex.  RWMutex is more
>>> complicated and therefore slower.  It's only useful to use an RWMutex
>>> when the read case is both contested and relatively slow.  If the read
>>> case is fast then the simpler Mutex will tend to be faster.  And then
>>> you don't have to worry about upgrading the lock.
>>> 
>>> Ian
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXNVFkc5H-L6K4Mt81gB6u91Ja07hob%3DS8Qwgy2buiQjQ%40mail.gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWJ%2BLPOoTk9H7bxAj8_dLsuhgOpy_bZZrGW%3D%2Bz6N%3DrX-w%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9AC5447A-2EAC-4FD0-9AEE-96EA517ABB94%40ix.netcom.com.

Re: [go-nuts] Upgradable RLock

[Robert Engels]

I don’t think that is true. A RW lock is always better when the reader activity 
is far greater than the writer - simply because in a good implementation the 
read lock can be acquired without blocking/scheduling activity. 

> On Jan 30, 2023, at 12:49 PM, Ian Lance Taylor  wrote:
> 
> On Sun, Jan 29, 2023 at 6:34 PM Diego Augusto Molina
>  wrote:
>> 
>> From times to times I write a scraper or some other tool that would 
>> authenticate to a service and then use the auth result to do stuff 
>> concurrently. But when auth expires, I need to synchronize all my goroutines 
>> and have a single one do the re-auth process, check the status, etc. and 
>> then arrange for all goroutines to go back to work using the new auth result.
>> 
>> To generalize the problem: multiple goroutines read a cached value that 
>> expires at some point. When it does, they all should block and some I/O 
>> operation has to be performed by a single goroutine to renew the cached 
>> value, then unblock all other goroutines and have them use the new value.
>> 
>> I solved this in the past in a number of ways: having a single goroutine 
>> that handles the cache by asking it for the value through a channel, using 
>> sync.Cond (which btw every time I decide to use I need to carefully re-read 
>> its docs and do lots of tests because I never get it right at first). But 
>> what I came to do lately is to implement an upgradable lock and have every 
>> goroutine do:
> 
> 
> We have historically rejected this kind of adjustable lock.  There is
> some previous discussion at https://go.dev/issue/4026,
> https://go.dev/issue/23513, https://go.dev/issue/38891,
> https://go.dev/issue/44049.
> 
> For a cache where checking that the cached value is valid (not stale)
> and fetching the cached value is quick, then in general you will be
> better off using a plain Mutex rather than RWMutex.  RWMutex is more
> complicated and therefore slower.  It's only useful to use an RWMutex
> when the read case is both contested and relatively slow.  If the read
> case is fast then the simpler Mutex will tend to be faster.  And then
> you don't have to worry about upgrading the lock.
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXNVFkc5H-L6K4Mt81gB6u91Ja07hob%3DS8Qwgy2buiQjQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CC51E5B9-B829-49D3-8421-7E19CA6570BC%40ix.netcom.com.

Re: [go-nuts] Creating and Linking to Shared Library Version of Go Runtime?

[Robert Engels]

Shared dynamic libraries do reduce the “wear and tear” on an SSD.  The binaries 
are loaded a single time and shared across processes - slightly longer startup 
times for the dynamic linkage. 

It is highly beneficial with large runtimes vs small standard library usage in 
tiny utilities.  

> On Jan 28, 2023, at 10:46 PM, Kurtis Rader  wrote:
> 
> 
> It does not surprise me that your shared run-time build takes more time than 
> the usual static build. The latter case is highly optimized while your use 
> case has probably been given very little consideration. I am also confused by 
> your argument for supporting linking against a shared Go runtime library. You 
> wrote earlier that the reason you want this is to "reduce ware (sic) and tear 
> on my SSD." I see no reason why linking against a shared Go run-time library 
> would reduce the "wear and tear" on your SSD. I think your premise is flawed.
> 
>> On Sat, Jan 28, 2023 at 8:29 PM jlfo...@berkeley.edu 
>>  wrote:
>> Thanks for the reply. I had mixed results.
>> 
>> On Fedora 37, go version go1.19.4 linux/amd64, in /usr/local/go/src as root 
>> I ran
>> 
>> go install -buildmode=shared std
>> 
>> That seemed to work.
>> 
>> In my normal working directory, as me, not root, I ran
>> 
>> go build -linkshared *.go
>> 
>> (I do a regular build in this directory by just running "go build *.go")
>> 
>> That resulted in a whole bunch of error messages of the style
>> 
>> go build internal/goarch: copying /tmp/go-build2481653269/b006/_pkg_.a: open 
>> /usr/local/go/pkg/linux_amd64_dynlink/internal/goarch.a: permission denied
>> 
>> So I became root and ran go build -linkshared *.go again.
>> 
>> This time it worked!! The result was 83584 byte binary, whereas the binary 
>> produced 
>> by the standard method is 4186764 bytes. That's a great savings!!! The small 
>> binary seemed
>> to work fine.
>> 
>> Just for yuks, I tried building my program as me again (not root). I got 
>> permission error messages
>> again, but now they look like
>> 
>> open /usr/local/go/pkg/linux_amd64_dynlink/archive/tar.shlibname: permission 
>> denied
>> 
>> There are 238 such lines.
>> 
>> There's another problem. Unlike what I would expect, it takes *longer* to 
>> build the shared version
>> than the static version.
>> 
>> As root
>> 
>> [root@localhost]# time go build *.go
>> 
>> real0m0.298s
>> user0m0.346s
>> sys 0m0.091s
>> 
>> [root@localhost]# time go build -linkshared *.go
>> 
>> real0m1.441s
>> user0m1.193s
>> sys 0m0.325s
>> 
>> That doesn't seem right.
>> 
>> Any advice?
>> 
>> Cordially,
>> Jon Forrest
>> 
>>> On Saturday, January 28, 2023 at 7:51:25 PM UTC-8 Ian Lance Taylor wrote:
>>> On Sat, Jan 28, 2023 at 11:27 AM jlfo...@berkeley.edu 
>>>  wrote: 
>>> > 
>>> > For people like me who have no intention of ever distributing a Go 
>>> > executable, I wonder 
>>> > if it would be useful, and possible, to link to a shared library version 
>>> > of the Go 
>>> > Runtime. This would make my binaries much smaller and would reduce ware 
>>> > and 
>>> > tear on my SSD. 
>>> > 
>>> > Of course, this presumes that such a shared library could be created in 
>>> > the first place. 
>>> > 
>>> > I did a quick Google and I didn't find this issue being previously 
>>> > discussed. 
>>> > 
>>> > Comments? 
>>> 
>>> At least on Linux systems it should work to run "go install 
>>> -buildmode=shared std" and then to build other Go programs with 
>>> "-linkshared". 
>>> 
>>> Ian 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/4dc94780-06e6-4aa3-a9b1-64b97dd85a5en%40googlegroups.com.
> 
> 
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-SBB%3DvFGYUGaXpf-8m6RH4HK4xEKjGUdMXMwjK90NMPg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6E3D7378-0994-44DE-A6E6-A71913D75EA1%40ix.netcom.com.

Re: [go-nuts] Variable Declaration inside infinite loop

[Robert Engels]

Java would no-op that entire loop since it does nothing that is externally 
visible. 

> On Jan 26, 2023, at 9:30 AM, Roland Müller  wrote:
> 
> 
> 
> Am Mittwoch, 25. Januar 2023 schrieb 'Jakob Borg' via golang-nuts 
> :
> > On 25 Jan 2023, at 12:26, Shashwat  wrote:
> >>
> >> Is it safe to declare a variable inside an infinite loop ?
> >>
> >> for {
> >> a := 0
> >> a += 1
> >> }
> >>
> >> Won't this cause memory allocation in every iteration ?
> >>
> >> As the declared variables will be allocated memory in the stack and not 
> >> heap memory, so garbage collector can't clear the allocated memory, and 
> >> eventually process will be terminated, isn't it?
> >
> > No. Apart from compiler optimisations, consider that the variable goes out 
> > of scope at the end of each iteration.
> >
> 
> In Java this may cause memory fragmentation when the amount of allocated 
> memory varies and the variables use much memory.
> 
> Dont know whether this may occur in Go too.
> 
> - Roland
> 
> > //jb
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to golang-nuts+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/golang-nuts/CE4EFF63-BD16-46F8-96EF-69000F7D9ACD%40kastelo.net.
> >
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CA%2B8p0G3nxAxWu0cNAJR_FWYQWvPdJHSP9v5X6VLhWFdh%2BxxenQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6467CA91-54F9-4856-99BB-D2B1F3E7A397%40ix.netcom.com.

Re: [go-nuts] Clarification of memory model behavior within a single goroutine

[Robert Engels]

The atomic functions force a memory barrier when atomically in conjunction with 
the atomic read of the same value. 

You could use CGO to call a C function to do what you desire - but it shouldn’t 
be necessary. 

Not sure what else I can tell you. 

> On Jan 22, 2023, at 8:12 PM, Peter Rabbitson  wrote:
> 
> 
>> On Mon, Jan 23, 2023 at 12:42 AM robert engels  wrote:
> 
>> Write data to memory mapped file/shared memory. Keep track of last written 
>> byte as new_length;
>> 
>> Use atomic.StoreUint64(pointer to header.length, new_length);
>> 
> 
> This does not answer the question I posed, which boils down to:
> 
> How does one insert the equivalent of smp_wmb() / asm volatile("" ::: 
> "memory") into a go program.
> 
> For instance is any of these an answer? 
> https://groups.google.com/g/golang-nuts/c/tnr0T_7tyDk/m/9T2BOvCkAQAJ
>  
>> readers read ...
> 
> Please don't focus on the reader ;) 
>  
>> This assumes you are always appending ,,, then it is much more complicated 
>> ... all readers have consumed the data before the writer reuses it.
> 
> Yes, it is much more complicated :) I am making a note to post the result 
> back to this thread in a few weeks when it is readable enough.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5F73A3C7-32C3-42A1-91A2-E1A0714FAEA5%40ix.netcom.com.

Re: [go-nuts] Clarification of memory model behavior within a single goroutine

[robert engels]

Write data to memory mapped file/shared memory. Keep track of last written byte 
as new_length;

Use atomic.StoreUint64(pointer to header.length, new_length);

readers read header.length atomically to determine the last valid byte (using 
whatever facilities their language has).

A reader then knows that bytes up to header.length are valid to consume.

This assumes you are always appending to the buffer - never reusing the earlier 
buffer space. If you desire to do this, then it is much more complicated as you 
need to determine that all readers have consumed the data before the writer 
reuses it.

The above must work in order for Go to have a happens before relationship with 
the atomics - all writes must be visible to a reader that see the updated value 
in the header.


> On Jan 22, 2023, at 12:53 PM, Peter Rabbitson  wrote:
> 
> 
> 
> On Sun, Jan 22, 2023 at 7:39 PM robert engels  <mailto:reng...@ix.netcom.com>> wrote:
> The atomic store will force a memory barrier - as long as the reader (in the 
> other process) atomically reads the “new value”, all other writes prior will 
> also be visible.
> 
> Could you translate this to specific go code? What would constitute what you 
> called "the atomic store" in the playground example I gave?  
>  
> BUT you can still have an inter-process race condition if you are updating 
> the same memory mapped file regions - and you need an OS mutex to protect 
> against this
> 
> Correct. This particular system is multiple-reader single-threaded-writer, 
> enforced by a Fcntl POSIX advisory lock. Therefore as long as I make the 
> specific writer consistent - I am done.
>  
> You can look at projects like https://github.com/OpenHFT/Chronicle-Queue 
> <https://github.com/OpenHFT/Chronicle-Queue> for ideas.
> 
> Still, large-scale shared memory systems are usually not required. I would 
> use a highly efficient message system like Nats.io <http://nats.io/> and not 
> reinvent the wheel. Messaging systems are also far more flexible.
> 
> 
> Nod, the example you linked is vaguely in line with what I want. You are also 
> correct that reinventing a wheel is bad form, and is to be avoided at all 
> costs. Yet the latency sensitivity of the particular IPC unfortunately does 
> call for an even rounder wheel. My problem isn't about "what to do" nor "is 
> there another way", but rather "how do I do this from within the confines of 
> go". 
>  
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAMrvTS%2BHqsqCOMay3c8D5LuTwcmtuZQJY7gs8Rw5rXBLiYwErg%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/CAMrvTS%2BHqsqCOMay3c8D5LuTwcmtuZQJY7gs8Rw5rXBLiYwErg%40mail.gmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/08AA8225-8F65-4297-AAB8-9FDA888C674B%40ix.netcom.com.