Re: [go-nuts] Opening network connections blocks system thread

[Kurtis Rader]

On Fri, Feb 9, 2024 at 2:10 PM Ian Lance Taylor  wrote:

> On Fri, Feb 9, 2024 at 11:57 AM Kurtis Rader  wrote:
> >
> > The connect() syscall is normally blocking. It doesn't return until the
> connection is established or an error occurs. It can be made non-blocking
> by putting the file-descriptor into non-blocking mode before the connect()
> call. However, that then requires either an async callback or another
> syscall to check whether the connection was established or an error
> occurred. Neither approach is idiomatic Go.
>
> That is true, but the Go standard library's net package does use
> non-blocking calls to connect internally when implementing net.Dial
> and friends.
>
> I don't have any problem running the original program on Linux 6.5.13.
>

Yes, I realized after my previous reply that Go could obviously use
non-blocking connect() calls coupled with select(), poll(), or similar
mechanisms to wakeup a goroutine blocked waiting for a net.Dial() (or
equivalent) connection to complete in order to minimize the number of OS
threads required to handle in-flight network connections. Without requiring
exposing an async callback or a mechanism to explicitly start a connection
and at a later time test whether it has been established or failed.

What might be happening for the O.P. is that the systems they are
connecting to are not explicitly accepting or rejecting the connections in
a timely manner. Thus causing a huge number of goroutines blocked waiting
for the net.Dial() to complete. The systems they are connecting to may be
simply discarding the TCP SYN packets due to firewall rules or something
similar. This is something that is going to be hard for the Go community to
provide help since it is fundamentally not an issue with Go itself.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD9j-4ikWnpz9kkd1stkj9sh9rR%3DGbSCbLp3wOo%2BCsXbuA%40mail.gmail.com.

Fwd: [go-nuts] [1.22] Is installing go under $HOME no longer possible?

[Johana de Roxas]

I just saw a big turn, that is the reason why to not do something about it.
And not even say about it. And to not work even on it.

-- Forwarded message -
From: Tetsuo Handa 
Date: Sat, Feb 10, 2024, 8:59 AM
Subject: [go-nuts] [1.22] Is installing go under $HOME no longer possible?
To: 


I noticed that go 1.22 started emitting

  warning: GOPATH set to GOROOT ($HOME/go) has no effect

and VSCode plugin noisily emits "Failed to run ..." messages
due to this warning message).

Although https://go.dev/doc/install suggests doing

  tar -C /usr/local -xzf go1.22.0.linux-amd64.tar.gz

and

  export PATH=$PATH:/usr/local/go/bin

, I'm doing

  tar -C $HOME -xzf go1.21.7.linux-amd64.tar.gz

and

  export PATH=$PATH:~/go/bin/

so that external packages can be installed without requiring
the write permission under /usr/local/go/ directory.

Installing under $HOME worked without problems until go 1.21.7 .
If this is an intentional change, please update https://go.dev/doc/install
to explicitly comment that installing under $HOME is not possible.

-- 
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/d2642225-3cf7-4c91-9059-fd45c880c6a5%40I-love.SAKURA.ne.jp
.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKZ8Z8XUzk4rHZTBqoUbM%3DD4WHDzZp16Rin8mr82sfa1Sq_EHA%40mail.gmail.com.

[go-nuts] Re: Profiling and PGO

['qiulaidongfeng' via golang-nuts]

Using benchmark and benchstat, you can compare the improvement obtained 
with PGO  .

On Friday, February 9, 2024 at 10:43:16 PM UTC+8 Agustin Horacio Urquiza 
Toledo wrote:

> I’m trying to compare the efficiency of two binaries before and after 
> applying Profile-Guided Optimization . For this, I’m using Go’s profiling, 
> specifically the flame graph. However, I’m getting two almost identical 
> stacks; the only difference is that some functions are inlined. It seems 
> that pprof doesn’t recognize that these functions are part of the same 
> execution path. I need to understand the percentage of improvement I’m 
> getting when compiling with PGO.
>
> My question is: Is there a way to indicate to pprof that these two paths 
> are part of the same execution? Or, is there a way to measure the 
> improvement obtained with PGO?"
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f8def38d-9f9a-409f-93c3-5c2c9f32c67bn%40googlegroups.com.

[go-nuts] [1.22] Is installing go under $HOME no longer possible?

[Tetsuo Handa]

I noticed that go 1.22 started emitting

  warning: GOPATH set to GOROOT ($HOME/go) has no effect

and VSCode plugin noisily emits "Failed to run ..." messages
due to this warning message).

Although https://go.dev/doc/install suggests doing

  tar -C /usr/local -xzf go1.22.0.linux-amd64.tar.gz

and

  export PATH=$PATH:/usr/local/go/bin

, I'm doing

  tar -C $HOME -xzf go1.21.7.linux-amd64.tar.gz

and

  export PATH=$PATH:~/go/bin/

so that external packages can be installed without requiring
the write permission under /usr/local/go/ directory.

Installing under $HOME worked without problems until go 1.21.7 .
If this is an intentional change, please update https://go.dev/doc/install
to explicitly comment that installing under $HOME is not possible.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d2642225-3cf7-4c91-9059-fd45c880c6a5%40I-love.SAKURA.ne.jp.

[go-nuts] Re: v1.22.0 Performance Issues

[Stephen Illingworth]

Hi Michael,

Thanks for looking at the profiles. I must confess that I didn't know about 
the --diff_base option. I'll take a closer look at the results of the diff 
and see if that can help me spot anything.

In answer to your questions: I'm certain these profiles were produced from 
the same source code and I'm also certain that the same amount of logical 
work was being done. I can rerun the tests and reproduce the results.

I'll raise an GitHub issue tomorrow if nothing else comes to light.


Thanks again
Stephen

On Friday 9 February 2024 at 22:16:10 UTC Michael Knyszek wrote:

> Hey Stephen,
>
> I poked at your profiles (diffing them) but my only guess is that it's a 
> compiler change that's the culprit. (For future reference, I produced this 
> output with the --diff_base flag, which subtracts one profile from 
> another.)
>
> Showing nodes accounting for -1.34s, 0.44% of 305.86s total
> Dropped 4 nodes (cum <= 1.53s)
> Showing top 20 nodes out of 248
>   flat  flat%   sum%cum   cum%
> -2.85s  0.93%  0.93% -2.89s  0.94%  
> github.com/jetsetilly/gopher2600/hardware/tia/audio.(*Audio).Step
> -2.80s  0.92%  1.85% -1.68s  0.55%  
> github.com/jetsetilly/gopher2600/hardware/tia.(*TIA).QuickStep
>  1.52s   0.5%  1.35%  1.91s  0.62%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*Video).Pixel
>  1.27s  0.42%  0.94%  2.04s  0.67%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*MissileSprite).tick
> -1.18s  0.39%  1.32% -1.18s  0.39%  
> github.com/jetsetilly/gopher2600/hardware/memory/cartridge.(*Cartridge).AccessPassive
>  
> (inline)
> -0.84s  0.27%  1.60% -0.99s  0.32%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*PlayerSprite).tick
>  0.67s  0.22%  1.38%  1.24s  0.41%  
> github.com/jetsetilly/gopher2600/hardware/cpu.(*CPU).read8Bit
>  0.61s   0.2%  1.18%  1.46s  0.48%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*Video).Tick
> -0.54s  0.18%  1.35% -0.59s  0.19%  
> github.com/jetsetilly/gopher2600/hardware/memory/cartridge.(*atari4k).Access
> -0.54s  0.18%  1.53% -0.51s  0.17%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*PlayerSprite).pixel
>  0.49s  0.16%  1.37%  0.49s  0.16%  
> github.com/jetsetilly/gopher2600/hardware/memory/memorymap.MapAddress 
> (inline)
>  0.47s  0.15%  1.22%  0.60s   0.2%  
> github.com/jetsetilly/gopher2600/hardware/memory/cartridge.(*Cartridge).Write 
> (inline)
>  0.47s  0.15%  1.06%  1.32s  0.43%  
> github.com/jetsetilly/gopher2600/hardware/riot/timer.(*Timer).Step
>  0.41s  0.13%  0.93%  0.41s  0.13%  
> github.com/jetsetilly/gopher2600/hardware/memory/vcs.(*RIOTMemory).ChipRefer
>  0.40s  0.13%   0.8%  0.44s  0.14%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*MissileSprite).pixel
>  0.39s  0.13%  0.67%  0.39s  0.13%  
> github.com/jetsetilly/gopher2600/hardware/memory/vcs.(*RIOTMemory).ChipHasChanged
>  0.36s  0.12%  0.55%  0.62s   0.2%  
> github.com/jetsetilly/gopher2600/hardware/riot/ports.(*Ports).Step
>  0.35s  0.11%  0.44%  0.35s  0.11%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*BallSprite).pixel
> -0.33s  0.11%  0.55% -0.59s  0.19%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*BallSprite).tick
>  0.33s  0.11%  0.44%  0.85s  0.28%  
> github.com/jetsetilly/gopher2600/hardware/tia/video.(*MissileSprite).tickHMOVE
>
> A couple questions just so we're on the same page:
> - Are you certain these two profiles were produced from the same source 
> code?
> - Are you certain these two profiles represent the same amount of logical 
> work being done? (As an example, in the context of a web service, I 
> wouldn't want to measure 30s of traffic, but rather N identical requests 
> handled to make an apples-to-apples comparison.)
>
> If yes to both, then I think my conclusion is that this is probably the 
> result of a compiler change, given that the flat time increases all seem to 
> come from "user" code (as opposed to, say, runtime code).
>
> Do you have a reproducer? It might be worth filing a GitHub issue at this 
> point. (https://github.com/golang/go/issues/new/choose)
>
> Thanks,
> Michael
>
> On Friday, February 9, 2024 at 3:13:12 PM UTC-5 Stephen Illingworth wrote:
>
>> Hello,
>>
>> I'm trying v1.22.0 and I'm measuring an approximate drop in performance 
>> of 10%. I'm comparing to v1.21.7
>>
>> I'm looking at the release notes but I can't see anything that I would 
>> think would be having an impact. Is anyone else noticing a performance drop?
>>
>> I've created a cpu.profile for my project for each the two go versions. 
>>
>> I can't see anything obviously different between the two profiles but 
>> then again, I'm not very skilled with pprof. I've uploaded them to github 
>> if anyone wants to help.
>>
>> https://github.com/JetSetIlly/gopher2600_performance_profiles
>>
>>
>> Regards
>> Stephen
>>
>

Re: [go-nuts] Opening network connections blocks system thread

[Ian Lance Taylor]

On Fri, Feb 9, 2024 at 11:57 AM Kurtis Rader  wrote:
>
> The connect() syscall is normally blocking. It doesn't return until the 
> connection is established or an error occurs. It can be made non-blocking by 
> putting the file-descriptor into non-blocking mode before the connect() call. 
> However, that then requires either an async callback or another syscall to 
> check whether the connection was established or an error occurred. Neither 
> approach is idiomatic Go.

That is true, but the Go standard library's net package does use
non-blocking calls to connect internally when implementing net.Dial
and friends.

I don't have any problem running the original program on Linux 6.5.13.

Ian


> On Fri, Feb 9, 2024 at 7:29 AM Venkat T V  wrote:
>>
>> Hi,
>>
>> I am debugging an issue where a server opening a large number of connections 
>> on startup sometimes dies with "program exceeds 1-thread limit". I know 
>> file IO operations could lock up an OS thread. Still seeing this crash after 
>> eliminating file IO, and it looks like "syscall.connect" is a blocking call 
>> and could tie up an OS thread. This is on Linux with golang 1.21.7.
>>
>> I wrote a small program to test this out. Running this with "go run 
>> osthreads.go -parallel 500 -threads 5" does trigger crashes sometimes, and I 
>> see goroutines blocked on "syscall.connect" and "syscall.fcntl". Could I get 
>> confirmation that this is expected behavior and Connect is a blocking 
>> syscall?
>>
>> ===
>> package main
>>
>> import (
>> "flag"
>> "fmt"
>> "net"
>> "runtime"
>> "runtime/debug"
>> "sync"
>> "time"
>> )
>>
>> func main() {
>> numThreads := flag.Int("threads", 10, "number of threads (in addition to 
>> GOMAXPROCS)")
>> parallelism := flag.Int("parallel", 100, "number of parallel goroutines to 
>> start")
>> flag.Parse()
>>
>> maxThreads := runtime.GOMAXPROCS(-1) + *numThreads
>> fmt.Printf("GOMAXPROCS=%d, max threads=%d\n", runtime.GOMAXPROCS(-1), 
>> maxThreads)
>> debug.SetMaxThreads(maxThreads)
>>
>> // Server that does not accept any connections
>> listener, err := net.Listen("tcp", "127.0.0.1:9090")
>> if err != nil {
>> fmt.Println(err)
>> return
>> }
>> defer listener.Close()
>>
>> wg := sync.WaitGroup{}
>> startSignal := make(chan struct{})
>>
>> // Spawn all goroutines
>> for i := 0; i < *parallelism; i++ {
>> wg.Add(1)
>> go func(id int) {
>> defer wg.Done()
>> <-startSignal
>>
>> conn, err := net.DialTimeout("tcp", "127.0.0.1:9090", time.Second)
>> if err != nil {
>> fmt.Printf("%d: error: %s\n", id, err)
>> return
>> }
>> defer conn.Close()
>> time.Sleep(time.Second)
>> }(i)
>> }
>>
>> time.Sleep(time.Second)
>>
>> // Start them all at once
>> close(startSignal)
>> wg.Wait()
>> }
>>
>> ===
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/33ab22bf-088a-4724-8cfb-62b7f51fca96n%40googlegroups.com.
>
>
>
> --
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD92p6ZBjugETbX%3D5KjCqS7HH-Dv4PSuUg%3D9AZwBbioG1A%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVWPysC2vsqvXeVWmwswNBEQ5uaKQkPsZLguuiDnY_Omw%40mail.gmail.com.

Re: [go-nuts] Is there a way to cast interface to embedded type?

[Mike Schinkel]

> On Feb 9, 2024, at 3:37 PM, Christopher C  
> wrote:
> 
> I have a base struct that implements an interface. There are multiple other 
> structs that embed this base struct.  I would like to pass the an interface 
> into a function that can cast it as the base struct and call some functions 
> tied to the base struct.
>  
> Something like this...
> https://go.dev/play/p/DUzXr31s8Pn

You can't cast like in your example, but you can create an interface — call is 
`Baser` using idiomatic interface naming to identify the `Base` type which will 
have an empty `Base()` method — and then type assert to it after which, if it 
succeeds you can type assert to `*Base`, like so:

https://go.dev/play/p/-gcKGf4_AFg 

Hope this helps.

-Mike

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1149B1AB-059E-42B4-A593-549D4793F7B5%40newclarity.net.

Re: [go-nuts] Is there a way to cast interface to embedded type?

[burak serdar]

On Fri, Feb 9, 2024 at 1:37 PM Christopher C
 wrote:
>
> I have a base struct that implements an interface. There are multiple other 
> structs that embed this base struct.  I would like to pass the an interface 
> into a function that can cast it as the base struct and call some functions 
> tied to the base struct.

No, because type assertion tests the type of the object contained in
an interface, and that object is not the base object.

However, you can put whatever functions you are planning to call in
that base object into another interface, and call that:

type Base struct {...}

func (Base) A() {}
func (Base) B() {}

type HasA interface {
  A()
}

func f(in HasA) {
   type hasB interface {
  B()
   }
   b,ok:=in.(hasB)
   if ok {
  b.B()
  }
}




>
> Something like this...
> https://go.dev/play/p/DUzXr31s8Pn
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/1ded992c-3f90-4c30-99a5-532e573cf16fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMV2Rqr2fhqDEOxD3jhe8U1fZ1GFvNoNRHanpAykfa8UVfFtyw%40mail.gmail.com.

[go-nuts] Is there a way to cast interface to embedded type?

[Christopher C]

I have a base struct that implements an interface. There are multiple other 
structs that embed this base struct.  I would like to pass the an interface 
into a function that can cast it as the base struct and call some functions 
tied to the base struct.
 
Something like this...
https://go.dev/play/p/DUzXr31s8Pn

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1ded992c-3f90-4c30-99a5-532e573cf16fn%40googlegroups.com.

[go-nuts] v1.22.0 Performance Issues

[Stephen Illingworth]

Hello,

I'm trying v1.22.0 and I'm measuring an approximate drop in performance of 
10%. I'm comparing to v1.21.7

I'm looking at the release notes but I can't see anything that I would 
think would be having an impact. Is anyone else noticing a performance drop?

I've created a cpu.profile for my project for each the two go versions. 

I can't see anything obviously different between the two profiles but then 
again, I'm not very skilled with pprof. I've uploaded them to github if 
anyone wants to help.

https://github.com/JetSetIlly/gopher2600_performance_profiles


Regards
Stephen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/eaf270fe-0010-48c5-9442-1a14e4de6b16n%40googlegroups.com.

Re: [go-nuts] Opening network connections blocks system thread

[Kurtis Rader]

The connect() syscall is normally blocking. It doesn't return until the
connection is established or an error occurs. It can be made
non-blocking by putting the file-descriptor into non-blocking mode before
the connect() call. However, that then requires either an async callback or
another syscall to check whether the connection was established or an error
occurred. Neither approach is idiomatic Go.

On Fri, Feb 9, 2024 at 7:29 AM Venkat T V  wrote:

> Hi,
>
> I am debugging an issue where a server opening a large number of
> connections on startup sometimes dies with "program exceeds 1-thread
> limit". I know file IO operations could lock up an OS thread. Still seeing
> this crash after eliminating file IO, and it looks like "syscall.connect"
> is a blocking call and could tie up an OS thread. This is on Linux with
> golang 1.21.7.
>
> I wrote a small program to test this out. Running this with "go run
> osthreads.go -parallel 500 -threads 5" does trigger crashes sometimes, and
> I see goroutines blocked on "syscall.connect" and "syscall.fcntl". Could I
> get confirmation that this is expected behavior and Connect is a blocking
> syscall?
>
> ===
> package main
>
> import (
> "flag"
> "fmt"
> "net"
> "runtime"
> "runtime/debug"
> "sync"
> "time"
> )
>
> func main() {
> numThreads := flag.Int("threads", 10, "number of threads (in addition to
> GOMAXPROCS)")
> parallelism := flag.Int("parallel", 100, "number of parallel goroutines
> to start")
> flag.Parse()
>
> maxThreads := runtime.GOMAXPROCS(-1) + *numThreads
> fmt.Printf("GOMAXPROCS=%d, max threads=%d\n", runtime.GOMAXPROCS(-1),
> maxThreads)
> debug.SetMaxThreads(maxThreads)
>
> // Server that does not accept any connections
> listener, err := net.Listen("tcp", "127.0.0.1:9090")
> if err != nil {
> fmt.Println(err)
> return
> }
> defer listener.Close()
>
> wg := sync.WaitGroup{}
> startSignal := make(chan struct{})
>
> // Spawn all goroutines
> for i := 0; i < *parallelism; i++ {
> wg.Add(1)
> go func(id int) {
> defer wg.Done()
> <-startSignal
>
> conn, err := net.DialTimeout("tcp", "127.0.0.1:9090", time.Second)
> if err != nil {
> fmt.Printf("%d: error: %s\n", id, err)
> return
> }
> defer conn.Close()
> time.Sleep(time.Second)
> }(i)
> }
>
> time.Sleep(time.Second)
>
> // Start them all at once
> close(startSignal)
> wg.Wait()
> }
>
> ===
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/33ab22bf-088a-4724-8cfb-62b7f51fca96n%40googlegroups.com
> 
> .
>


-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD92p6ZBjugETbX%3D5KjCqS7HH-Dv4PSuUg%3D9AZwBbioG1A%40mail.gmail.com.

[go-nuts] Opening network connections blocks system thread

[Venkat T V]

Hi,

I am debugging an issue where a server opening a large number of 
connections on startup sometimes dies with "program exceeds 1-thread 
limit". I know file IO operations could lock up an OS thread. Still seeing 
this crash after eliminating file IO, and it looks like "syscall.connect" 
is a blocking call and could tie up an OS thread. This is on Linux with 
golang 1.21.7.

I wrote a small program to test this out. Running this with "go run 
osthreads.go -parallel 500 -threads 5" does trigger crashes sometimes, and 
I see goroutines blocked on "syscall.connect" and "syscall.fcntl". Could I 
get confirmation that this is expected behavior and Connect is a blocking 
syscall?

===
package main

import (
"flag"
"fmt"
"net"
"runtime"
"runtime/debug"
"sync"
"time"
)

func main() {
numThreads := flag.Int("threads", 10, "number of threads (in addition to 
GOMAXPROCS)")
parallelism := flag.Int("parallel", 100, "number of parallel goroutines to 
start")
flag.Parse()

maxThreads := runtime.GOMAXPROCS(-1) + *numThreads
fmt.Printf("GOMAXPROCS=%d, max threads=%d\n", runtime.GOMAXPROCS(-1), 
maxThreads)
debug.SetMaxThreads(maxThreads)

// Server that does not accept any connections
listener, err := net.Listen("tcp", "127.0.0.1:9090")
if err != nil {
fmt.Println(err)
return
}
defer listener.Close()

wg := sync.WaitGroup{}
startSignal := make(chan struct{})

// Spawn all goroutines
for i := 0; i < *parallelism; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
<-startSignal

conn, err := net.DialTimeout("tcp", "127.0.0.1:9090", time.Second)
if err != nil {
fmt.Printf("%d: error: %s\n", id, err)
return
}
defer conn.Close()
time.Sleep(time.Second)
}(i)
}

time.Sleep(time.Second)

// Start them all at once
close(startSignal)
wg.Wait()
}

===

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/33ab22bf-088a-4724-8cfb-62b7f51fca96n%40googlegroups.com.

[go-nuts] Re: Go 1.22 binary size improvements?

['Brian Candler' via golang-nuts]

Sorry you're right, it's "otool -L" for macOS.

And you say that if you cross-compile from Linux with 1.22.0 to build a 
macOS binary, you get a size approx 93.8MB as well? Very odd.

On Friday 9 February 2024 at 14:43:35 UTC cpu...@gmail.com wrote:

> Yes, CGO is disabled. ldd doesn't seem to exist on MacOS:
>
> make build && ls -la ./evcc && otool -L ./evcc
>
> 1.21.7
>
> CGO_ENABLED=0 go build -v -tags=release -trimpath -ldflags='-X 
> github.com/evcc-io/evcc/server.Version=0.124.1 -X 
> github.com/evcc-io/evcc/server.Commit=596071b42 -s -w'
> -rwxr-xr-x  1 andig  staff  *93824418*  9 Feb 14:38 ./evcc
> ./evcc:
> /usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current 
> version 0.0.0)
> /usr/lib/libresolv.9.dylib (compatibility version 0.0.0, current 
> version 0.0.0)
> 
> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation 
> (compatibility version 0.0.0, current version 0.0.0)
> /System/Library/Frameworks/Security.framework/Versions/A/Security 
> (compatibility version 0.0.0, current version 0.0.0)
>
> 1.22.0
>
> CGO_ENABLED=0 go build -v -tags=release -trimpath -ldflags='-X 
> github.com/evcc-io/evcc/server.Version=0.124.1 -X 
> github.com/evcc-io/evcc/server.Commit=596071b42 -s -w'
> -rwxr-xr-x  1 andig  staff  *81119026*  9 Feb 14:37 ./evcc
> ./evcc:
> /usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current 
> version 0.0.0)
> /usr/lib/libresolv.9.dylib (compatibility version 0.0.0, current 
> version 0.0.0)
> 
> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation 
> (compatibility version 0.0.0, current version 0.0.0)
> /System/Library/Frameworks/Security.framework/Versions/A/Security 
> (compatibility version 0.0.0, current version 0.0.0)
>
> On Friday, February 9, 2024 at 2:22:36 PM UTC+1 Brian Candler wrote:
>
>> Have you at any point set CGO_ENABLED=0 ?
>>
>> What does "ldd /path/to/binary" show on both the old (larger) and new 
>> (smaller) binaries?  Maybe one is dynamically linked and the other 
>> statically linked?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/510198df-4b95-4c55-8cc1-a40c13703578n%40googlegroups.com.

[go-nuts] Re: Go 1.22 binary size improvements?

[cpu...@gmail.com]

Yes, CGO is disabled. ldd doesn't seem to exist on MacOS:

make build && ls -la ./evcc && otool -L ./evcc

1.21.7

CGO_ENABLED=0 go build -v -tags=release -trimpath -ldflags='-X 
github.com/evcc-io/evcc/server.Version=0.124.1 -X 
github.com/evcc-io/evcc/server.Commit=596071b42 -s -w'
-rwxr-xr-x  1 andig  staff  *93824418*  9 Feb 14:38 ./evcc
./evcc:
/usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current 
version 0.0.0)
/usr/lib/libresolv.9.dylib (compatibility version 0.0.0, current 
version 0.0.0)

/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation 
(compatibility version 0.0.0, current version 0.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security 
(compatibility version 0.0.0, current version 0.0.0)

1.22.0

CGO_ENABLED=0 go build -v -tags=release -trimpath -ldflags='-X 
github.com/evcc-io/evcc/server.Version=0.124.1 -X 
github.com/evcc-io/evcc/server.Commit=596071b42 -s -w'
-rwxr-xr-x  1 andig  staff  *81119026*  9 Feb 14:37 ./evcc
./evcc:
/usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current 
version 0.0.0)
/usr/lib/libresolv.9.dylib (compatibility version 0.0.0, current 
version 0.0.0)

/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation 
(compatibility version 0.0.0, current version 0.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security 
(compatibility version 0.0.0, current version 0.0.0)

On Friday, February 9, 2024 at 2:22:36 PM UTC+1 Brian Candler wrote:

> Have you at any point set CGO_ENABLED=0 ?
>
> What does "ldd /path/to/binary" show on both the old (larger) and new 
> (smaller) binaries?  Maybe one is dynamically linked and the other 
> statically linked?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5787a122-8581-41e7-81d5-9ae2b0e28942n%40googlegroups.com.

[go-nuts] Profiling and PGO

[Agustin Horacio Urquiza Toledo]

I’m trying to compare the efficiency of two binaries before and after 
applying Profile-Guided Optimization . For this, I’m using Go’s profiling, 
specifically the flame graph. However, I’m getting two almost identical 
stacks; the only difference is that some functions are inlined. It seems 
that pprof doesn’t recognize that these functions are part of the same 
execution path. I need to understand the percentage of improvement I’m 
getting when compiling with PGO.

My question is: Is there a way to indicate to pprof that these two paths 
are part of the same execution? Or, is there a way to measure the 
improvement obtained with PGO?"

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/11628d55-73c3-44e1-91bf-8fcc8adb1631n%40googlegroups.com.

[go-nuts] Re: Go 1.22 binary size improvements?

['Brian Candler' via golang-nuts]

Have you at any point set CGO_ENABLED=0 ?

What does "ldd /path/to/binary" show on both the old (larger) and new 
(smaller) binaries?  Maybe one is dynamically linked and the other 
statically linked?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/fb7d0cf7-0ebf-4205-aaac-cc8422fab2cfn%40googlegroups.com.

[go-nuts] Re: Go 1.22 binary size improvements?

[cpu...@gmail.com]

Just gave it another try cross-compiling to Arm on Linux. This time binary 
size stays the same. Sorry if my excitement was too premature.

On Friday, February 9, 2024 at 2:11:25 PM UTC+1 cpu...@gmail.com wrote:

> It's really surprising (positively). Compiling with -ldflags='-s -w' if 
> anyone wants to try this.
>
> On Friday, February 9, 2024 at 1:23:20 PM UTC+1 qiulaidongfeng wrote:
>
>> Quick search to find go1.22 as follows  CL  like  
>> https://go-review.googlesource.com/c/go/+/521615   reduces binary size. 
>> However, from 93MB to 81MB, I think it is most likely the result of the 
>> joint efforts of many CLs.
>>
>> On Friday, February 9, 2024 at 7:46:48 PM UTC+8 cpu...@gmail.com wrote:
>>
>>> I was surprised to see my application (MacOS binary) go from 93MB to 
>>> 81MB (no PGO). Release notes don't mention this.
>>>
>>> Has anyone made similar experience or knows why that is?
>>>
>>> Thanks,
>>> Andi
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/74abf623-f1cc-4835-9893-8bc91d3e2afbn%40googlegroups.com.

[go-nuts] Re: Go 1.22 binary size improvements?

[cpu...@gmail.com]

It's really surprising (positively). Compiling with -ldflags='-s -w' if 
anyone wants to try this.

On Friday, February 9, 2024 at 1:23:20 PM UTC+1 qiulaidongfeng wrote:

> Quick search to find go1.22 as follows  CL  like  
> https://go-review.googlesource.com/c/go/+/521615   reduces binary size. 
> However, from 93MB to 81MB, I think it is most likely the result of the 
> joint efforts of many CLs.
>
> On Friday, February 9, 2024 at 7:46:48 PM UTC+8 cpu...@gmail.com wrote:
>
>> I was surprised to see my application (MacOS binary) go from 93MB to 81MB 
>> (no PGO). Release notes don't mention this.
>>
>> Has anyone made similar experience or knows why that is?
>>
>> Thanks,
>> Andi
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c93a73d1-ea18-4d12-9127-0787ffb5f51cn%40googlegroups.com.

[go-nuts] Re: Go 1.22 binary size improvements?

['qiulaidongfeng' via golang-nuts]

Quick search to find go1.22 as follows  CL  like  
https://go-review.googlesource.com/c/go/+/521615   reduces binary size. 
However, from 93MB to 81MB, I think it is most likely the result of the 
joint efforts of many CLs.

On Friday, February 9, 2024 at 7:46:48 PM UTC+8 cpu...@gmail.com wrote:

> I was surprised to see my application (MacOS binary) go from 93MB to 81MB 
> (no PGO). Release notes don't mention this.
>
> Has anyone made similar experience or knows why that is?
>
> Thanks,
> Andi
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f7d074ec-88ad-44fe-a472-811663bd1199n%40googlegroups.com.

[go-nuts] Go 1.22 binary size improvements?

[cpu...@gmail.com]

I was surprised to see my application (MacOS binary) go from 93MB to 81MB 
(no PGO). Release notes don't mention this.

Has anyone made similar experience or knows why that is?

Thanks,
Andi

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d9ba2568-90d7-4a0f-8b25-83a3f01ad662n%40googlegroups.com.

[go-nuts] What are the reason for the performance improvement cause by alloc data of different sizes to adjacent memory addresses?

['qiulaidongfeng' via golang-nuts]

I recently added a new API (AllocSoonUse) to the 
https://gitee.com/qiulaidongfeng/arena I wrote (copy in 
https://github.com/qiulaidongfeng/arena)
This API allocdata to adjacent []byte, and the length can be set.
I use this benchmark code : https://go.dev/play/p/nAGveraT5_h

benchmark Brief description, use AllocSoonUse and new to allocate some data 
of different sizes that does not contain Pointers and use it soon after 
allocation.

I use `go test -bench=. -count=10 -benchtime=2s > new.txt && benchstat 
new.txt`  got this result
 │   new.txt   │
 │   sec/op│
NoStd-16   207.4µ ± 4%
Std-16 245.0µ ± 1%
geomean225.4µ

It appears that using AllocSoonUse takes 15.34% less time than using new in 
some scenarios.

I initially speculated that the reason for the performance improvement was 
the change in cache hit rate.

But using this benchmark https://go.dev/play/p/SCNMV6I9GWC and `go test 
-bench=. -count=10 -benchtime=2s > new.txt && benchstat new.txt`  got this 
result
 │   new.txt   │
 │   sec/op│
NoStd-16   206.6µ ± 3%
Std-16 236.0µ ± 2%
geomean220.9µ

It appears that there is still a 12.45% performance gap after commenting 
out the access code to allocated memory to reduce the access to memory.

I want to get the performance gap so the reason.

The possibility has been ruled out, AllocSoonUse  allows free memory in a 
way that allows for immediate reuse  , but I did not enable this opt-in.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/13dfb086-3fc4-4361-80eb-fc6466cfd72fn%40googlegroups.com.