Re: [go-nuts] Why is the error sent to original channel can still be received after recreating the channel?

[robert engels]

Because the channel is captured by reference, so when the Go routine runs it 
already has the later value.

Your code has a race condition - run it under the race detector.

> On Sep 19, 2024, at 11:30 AM, Feng Liyuan (SunRunAway) 
>  wrote:
> 
> I have a piece of Go code where I create a buffered error channel and send an 
> error to it from a goroutine. Then, in the main, I recreate this channel. To 
> my confusion, even after recreating the channel, I still receive the error 
> sent by the original goroutine. Here is the code:
> 
> https://go.dev/play/p/y3roOPKO0iy
> 
> 
> Can someone help explain why this happens?
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3a49b432-a8fa-4bb3-bcd2-a8fcd679a0c3n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A10FA8D9-B5DF-4F56-B37C-5E21A2C5B591%40ix.netcom.com.

Re: [go-nuts] io.Copy(conn, os.Stdin) do not work as expected

[Robert Engels]

Also, it would be simpler to not run the copy in a go routine until you verify that is working as expected. Then use wait groups to control the go routine lifecycle. On Sep 19, 2024, at 8:51 AM, Robert Engels  wrote:Are you certain you are running it in a terminal that has stdin available for reading?On Sep 19, 2024, at 8:08 AM, 李世民  wrote:The details as following link:https://stackoverflow.com/questions/78996496/io-copyconn-os-stdin-do-not-work-as-expected#comment139289803_78996496Please help me identify where my problem is.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c1efd91e-2815-4a6c-b450-5d47423131c4n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/59D3B652-30F5-43B7-AA52-76D1113C9BFE%40ix.netcom.com.

Re: [go-nuts] io.Copy(conn, os.Stdin) do not work as expected

[Robert Engels]

Are you certain you are running it in a terminal that has stdin available for reading?On Sep 19, 2024, at 8:08 AM, 李世民  wrote:The details as following link:https://stackoverflow.com/questions/78996496/io-copyconn-os-stdin-do-not-work-as-expected#comment139289803_78996496Please help me identify where my problem is.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c1efd91e-2815-4a6c-b450-5d47423131c4n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/E844BD5C-3543-47EC-8833-4C91A901AA93%40ix.netcom.com.

Re: [go-nuts] Re: Can we suggest to the GC that I think this memory can be reclaimed?

[Robert Engels]

With Go there is no possibility of use after free unless you use unsafe calls. On Sep 16, 2024, at 10:47 AM, Kevin Chowski  wrote:In particular, since Go has a goal of avoiding use-after-free bugs, the GC would have to do the same validation it is doing anyway to double-check that the caller is not wrong.On Saturday, September 14, 2024 at 10:55:49 AM UTC-6 Frederik Zipp wrote:Yes, by no longer holding a reference to it. If an object is no longer referenced, it becomes a candidate for garbage collection.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20d2db53-fa13-44e3-9e8f-a545078e85b8n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CE46D674-162A-4AE2-B171-4CA0B28F006C%40ix.netcom.com.

Re: [go-nuts] http resp.Write & httputil.DumpResponse include extra text with body

[Robert Engels]

If you set the content length in the header it should turn off chunked encoding. On Sep 13, 2024, at 8:38 AM, Peter Galbavy  wrote:Makes sense and good to know, but for diag output is there any way to turn it off?On Friday 13 September 2024 at 14:25:19 UTC+1 Eli Lindsey wrote:Those are sizes from chunked encoding. -eli On Sep 13, 2024, at 9:10 AM, Peter Galbavy  wrote:I am trying to add some low level tracing to a client app to validate the responses coming from a SaaS platform.When I use either http's resp.Write(os.Stderr) or httputil.DumpResponse(resp, true) I see a hex content length immediately before the body and a literal "0" on a line after,i.e. using a local copy of the doc example for DumpResponse() but with a real URL and outputting "string(dump)" instead of quotes bytes) to a local server I get (HTML between open/close removed for brevity):---HTTP/1.1 200 OKTransfer-Encoding: chunkedConnection: keep-aliveContent-Type: text/htmlDate: Fri, 13 Sep 2024 13:05:33 GMTEtag: W/"62136b7a-d2"Last-Modified: Mon, 21 Feb 2022 10:37:46 GMTServer: nginx/1.27.0d2...0---The "d2" is the content length and the "0" is extra. Compared against Postman and curl on the command line, to be sure.Is this expected by undocumented or have I missed something please?



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/07d82e3f-5f14-40da-8856-602c1784e986n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/30b85162-38b5-47a4-8d1a-432ef3d0ea34n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/EC665F51-C7B8-4E1D-8A7A-16CE0D0D0AC0%40ix.netcom.com.

Re: [go-nuts] Dereferencing struct field triggers heap allocation for the complete struct

[Robert Engels]

Yes, but the called function can retain a reference to the string - strings are pointers. it cannot retain a reference to the scalar. I’m guessing if any of the elements can be retained then the entire structure is allocated on the heap - I don’t think it has to but it is probably the current implementation. On Aug 30, 2024, at 5:34 PM, rkerno  wrote:Thanks Robert.  I don't think it's the value that's causing thisI still get the allocation if I remove the value from the equation.  It's something to do with the root level having children.The escape analysis from the refactored code below is:go build -gcflags='-m=2' ../alloc.go:12:27: parameter y leaks to {heap} with derefs=0:./alloc.go:12:27:   flow: {heap} = y:./alloc.go:12:27:     from y.key (dot) at ./alloc.go:19:6./alloc.go:12:27:     from fn(y.key) (call parameter) at ./alloc.go:19:4Strings can't reference anything, AFAIK.BenchmarkAllocations0-2         97209274                28.76 ns/op            0 B/op          0 allocs/opBenchmarkAllocations1-2          1395814               796.9 ns/op          1152 B/op          1 allocs/optype (	Y struct {		key      string		children []Y		unused   [128]uint64 // Demonstrates that the whole struct is placed on the heap!	}	useKeyFunc = func(key string))func scanY(fn useKeyFunc, y Y) {	if y.children != nil {		for i := 0; i < len(y.children); i++ {			scanY(fn, y.children[i])		}		return	}	fn(y.key)}func printKey(key string) {}func Benchmark_ZeroAllocations(b *testing.B) {	b.ReportAllocs()	b.ResetTimer()	b.RunParallel(func(pb *testing.PB) {		for pb.Next() {			scanY(printKey,Y{	key:      "root",	children: []Y{},},			)		}	})}func Benchmark_OneAllocation(b *testing.B) {	b.ReportAllocs()	b.ResetTimer()	b.RunParallel(func(pb *testing.PB) {		for pb.Next() {			scanY(printKey,Y{	key: "root",	children: []Y{		{			key:      "level1",			children: nil,		},	},},			)		}	})}On Saturday, August 31, 2024 at 5:02:28 AM UTC+12 robert engels wrote:because when you only access an int that is passed by value in the function it knows it can’t escape.if you pass the string/value than the the variable can escape, and since the value could point back to the struct itself, it can escape, meaning the entire struct needs to be on the heap.On Aug 30, 2024, at 3:43 AM, rkerno <rik.ke...@projectcatalysts.com> wrote:Hi Everyone,https://gist.github.com/rkerno/c875609bdeb2459582609da36b54bf72I'm struggling to wrap my head around the root cause of this issue.Accessing certain fields of a struct triggers go to allocate the complete struct on the heap.  The key ingredients are:A struct with a contained slice of structs defined inlineCalls to functions via a function pointerWhat follows is a contrived example to demonstrate / investigate the problem.const (	//	// When requireZeroAllocations == true...	//	//     BenchmarkAllocations-2          27763290                52.28 ns/op          0 B/op          0 allocs/op	//	// When requireZeroAllocations == false...	//	//     BenchmarkAllocations-2          14403922               1532.00 ns/op      2304 B/op          2 allocs/op	//	requireZeroAllocations = false)type (	X struct {		id       uint		key      string		value    any		children []X		useId    useIdFunc		useX     useXFunc		useKey   useKeyFunc		useValue useValueFunc		unused   [128]uint64 // Demonstrates that the whole struct is placed on the heap!	}	useIdFunc    = func(id uint)	useXFunc     = func(key string, value any)	useKeyFunc   = func(key string)	useValueFunc = func(value any))func scanX(x X) {	if len(x.children) > 0 {		for i := 0; i < len(x.children); i++ {			scanX(x.children[i])		}		return	}	x.useId(x.id)	if !requireZeroAllocations {		// Why can we access the id field without an allocation, but as soon as we access		// the string or any fields the entire struct is placed on the heap?		x.useKey(x.key)		x.useX(x.key, x.value)		x.useValue(x.value)	}}func printId(id uint) {}func printX(key string, value any) {}func printKey(key string) {}func printValue(value any) {}The benchmark used to demonstrate the behaviour is:func BenchmarkAllocations(b *testing.B) {	b.ReportAllocs()	b.ResetTimer()	b.RunParallel(func(pb *testing.PB) {		for pb.Next() {			scanX(X{	id:    0,	key:   "root",	value: nil,	children: []X{		{			id:    1,			key:   "level1",			value: nil,			children: []X{{	id:       2,	key:      "k1",	value:    "v1",	children: nil,	useId:    printId,	useX:     printX,	useKey:   printKey,	useValue: printValue,},			},		},	},},			)		}	})}Any insights or ideas why go exhibits this behaviour would be appreciated.Cheers,Rik

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe 

Re: [go-nuts] Dereferencing struct field triggers heap allocation for the complete struct

[robert engels]

because when you only access an int that is passed by value in the function it 
knows it can’t escape.

if you pass the string/value than the the variable can escape, and since the 
value could point back to the struct itself, it can escape, meaning the entire 
struct needs to be on the heap.


> On Aug 30, 2024, at 3:43 AM, rkerno  
> wrote:
> 
> Hi Everyone,
> 
> https://gist.github.com/rkerno/c875609bdeb2459582609da36b54bf72 
> 
> 
> I'm struggling to wrap my head around the root cause of this issue.
> 
> Accessing certain fields of a struct triggers go to allocate the complete 
> struct on the heap.  The key ingredients are:
> A struct with a contained slice of structs defined inline
> Calls to functions via a function pointer
> What follows is a contrived example to demonstrate / investigate the problem.
> 
> const (
>   //
>   // When requireZeroAllocations == true...
>   //
>   // BenchmarkAllocations-2  2776329052.28 
> ns/op  0 B/op  0 allocs/op
>   //
>   // When requireZeroAllocations == false...
>   //
>   // BenchmarkAllocations-2  14403922   1532.00 
> ns/op  2304 B/op  2 allocs/op
>   //
>   requireZeroAllocations = false
> )
> 
> type (
>   X struct {
>   id   uint
>   key  string
>   valueany
>   children []X
>   useIduseIdFunc
>   useX useXFunc
>   useKey   useKeyFunc
>   useValue useValueFunc
>   unused   [128]uint64 // Demonstrates that the whole struct is 
> placed on the heap!
>   }
>   useIdFunc= func(id uint)
>   useXFunc = func(key string, value any)
>   useKeyFunc   = func(key string)
>   useValueFunc = func(value any)
> )
> 
> func scanX(x X) {
>   if len(x.children) > 0 {
>   for i := 0; i < len(x.children); i++ {
>   scanX(x.children[i])
>   }
>   return
>   }
> 
>   x.useId(x.id)
>   if !requireZeroAllocations {
>   // Why can we access the id field without an allocation, but as 
> soon as we access
>   // the string or any fields the entire struct is placed on the 
> heap?
>   x.useKey(x.key)
>   x.useX(x.key, x.value)
>   x.useValue(x.value)
>   }
> }
> 
> func printId(id uint) {
> }
> func printX(key string, value any) {
> }
> func printKey(key string) {
> }
> func printValue(value any) {
> }
> 
> 
> The benchmark used to demonstrate the behaviour is:
> 
> func BenchmarkAllocations(b *testing.B) {
>   b.ReportAllocs()
>   b.ResetTimer()
>   b.RunParallel(func(pb *testing.PB) {
>   for pb.Next() {
>   scanX(
>   X{
>   id:0,
>   key:   "root",
>   value: nil,
>   children: []X{
>   {
>   id:1,
>   key:   "level1",
>   value: nil,
>   children: []X{
>   {
>   id: 
>   2,
>   key:
>   "k1",
>   value:  
>   "v1",
>   
> children: nil,
>   useId:  
>   printId,
>   useX:   
>   printX,
>   useKey: 
>   printKey,
>   
> useValue: printValue,
>   },
>   },
>   },
>   },
>   },
>   )
>   }
>   })
> }
> 
> Any insights or ideas why go exhibits this behaviour would be appreciated.
> 
> Cheers,
> Rik
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To vi

Re: [go-nuts] Connectivity breakage from removal of TLS RSA KEX from default encryption suite

[robert engels]

The statement "Especially that of a previously compiled program will run under 
a new compiler version with the same behaviour” is not true with many 
platforms. Even with Java, with a great backwards guarantee, this is not the 
case (i.e. running on a later JVM) - although this is a somewhat recent 
behavioral change.

I said to add a “breaking changes” to the release notes. If you are upgrading 
your compiler/stdlib version, and rebuilding the software, it would be the 
builders responsibility to check that.

Even if the end user knew it was a Go program, they probably can’t build it, 
and/or would not know how to add runtime options to work around it.

> On Aug 27, 2024, at 11:06 AM, Creaky  wrote:
> 
> 
> 
> On Tuesday, August 27, 2024 at 8:00:27 AM UTC+10 robert engels wrote:
> I don’t think being concerned about the user is correct - I suspect that many 
> (most?) of the users have no idea the program is even written in Go… 
> 
> This is a builders responsibility imo to either fail at start-up with a user 
> understandable error, add mitigating code, or don’t build and release using 
> the latest release. 
> 
> > On Aug 26, 2024, at 4:45 PM, Ian Lance Taylor  > > wrote: 
> > 
> > On Mon, Aug 26, 2024 at 2:28 PM robert engels  > > wrote: 
> >> 
> >> Isn’t the version release notes 
> >> shttps://pkg.go.dev/net/http#Request.Hostufficient 
> >> <http://pkg.go.dev/net/http#Request.Hostufficient>? Maybe add a "breaking 
> >> changes" section? https://tip.golang.org/doc/go1.23 
> >> <https://tip.golang.org/doc/go1.23> 
> >
> 
> The Go ecosystem has created the expectation that a compiled program will run 
> correctly. Especially that of a previously compiled program will run under a 
> new compiler version with the same behaviour.
> 
> Breaking changes that change the contract of behaviour continuity needs 
> special mentioning and highlighting. Changing defaults is one such change.
> 
> A builder may not be cognisant of the significance or impact of a release 
> note entry. Especially when the breaking behaviour is not noted or requires 
> extensive understanding of a specialist technical area.
>  
> The users impacted (or I prefer audience set) can be broad. From programmers, 
> deployers, administrators to end users. A good portion of the audience set 
> will have the capability to know it is a Go program at fault.
>  
> The key is discoverability of the issue's cause and fix. Good documentation 
> and clear communications aids in this.
> 
> For breaking changes, the audience needs the ability to find a solution and 
> communication needs to support easy discovery of said solution.
> 
> The current single line Release note entry is insufficient for meeting the 
> need of easy discovery.
> 
> Practices I have seen to address discoverability includes identifying 
> breaking changes separately, often in its own document. Usually with a link 
> to an accompanying technical note, bulletin or blog. 
> 
> The accompanying document then describes the change in detail including:
> Summary of what the change is.
> Reason for change.
> Impact of change, what breaks and how to identify it.
> Various solutions or fixes to restore behaviour. With source code or just 
> with executable.
> Commitment to support or provide continuity for said solutions or fixes.
> Increasing the easy of discovery helps everyone in the ecosystem.
> 
> Regards,
> 
> Creaky
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e728d646-3baa-4d26-a396-6cb741b864b5n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/e728d646-3baa-4d26-a396-6cb741b864b5n%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/AD552B33-6DA5-40D0-8276-5098AEAB38EF%40ix.netcom.com.

Re: [go-nuts] Connectivity breakage from removal of TLS RSA KEX from default encryption suite

[robert engels]

Hi,

Isn’t the version release notes sufficient? Maybe add a "breaking changes" 
section? https://tip.golang.org/doc/go1.23

> On Aug 26, 2024, at 4:20 PM, Ian Lance Taylor  wrote:
> 
> On Mon, Aug 26, 2024 at 8:31 AM Creaky  wrote:
>> 
>> Want to discuss the impacts from the implementation of the proposal 
>> crypto/tls: disable RSA key exchange cipher suites by default #63413 and 
>> possible ways forward beyond go change some code (whilst it may fix the 
>> issue in the short term, has certain drawbacks, takes time, and is 
>> impractical in some cases).
>> 
>> This proposal was introduced in Go Version 1.22 for crypto/tls and brought a 
>> significant behavioural change impacting client server connectivity with 
>> very little announcement.
>> 
>> People are seeing TLS handshake failures causing complete failure of 
>> connectivity and not knowing where the issue is. As tooling starts to take 
>> on the 1.22 / 1.23 Golang changes, the impacts from this change are now 
>> bubbling to the surface.
>> 
>> I can see why the motivation behind the change is from a good place. There 
>> are weaknesses in using TLS with RSA encrypted key exchange. However, the 
>> change as implemented is producing poor outcomes.
>> 
>> From a high level perspective, it is far better to protect communication 
>> with possibly vulnerable encryption than being completely in the clear. 
>> However this breakage is encouraging some people to revert to in clear 
>> connectivity for lacking (unknowing) a better option.
>> 
>> Reading through the original proposal it appears considerations of change 
>> impact narrowly focused on Internet connected clients and servers and did 
>> not consider the various audience groups impacted by the proposed (and now 
>> accepted) change.
>> 
>> This change is now causing one of the more costly failure types, that being 
>> in the field failures.
>> 
>> What can be done now this proposal is implemented?
>> 
>> Can the proposal and its implementation be reversed? Or removed and 
>> reintroduced with better communication?
> 
> Thanks for the detailed note.
> 
> My takeaway here is that the Go team needs to provide better
> communication about the consequences that people will see from this
> kind of change, and how to better communicate those consequences and
> how to work around them.  That is, I believe that the change is a good
> one for the overall ecosystem.  But people can encounter this change
> and not know what caused it and not know the relatively simple
> workarounds that are available.
> 
> As you know, those workarounds are described at
> https://go.dev/doc/godebug.  People working with an executable program
> can set the GODEBUG variable in the environment.  People developing a
> program from source can do that, and can also add a go:debug directive
> to the main package and/or a godebug setting in a go.mod or go.work
> file.  There are no plans to remove the tlsrsakex setting.
> 
> Still you're absolutely right that the average user will see some sort
> of failure to communicate, and will have no clear way to translate
> that into what they need to do to change.  So the question is: how can
> we communicate that better?  What are the right channels?  Would a Go
> blog entry be sufficient?  Also, what kinds of errors will users see
> in practice, and how can we get those errors to direct them to the
> GODEBUG setting?
> 
> Thanks.
> 
> Ian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWTRwuFjZh_FEEAnJATS_io1VxP5c%2BS4Dv_oeshX-%2B0qA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/59ECA4AB-CD15-4BA7-BFAC-540804A7D1C4%40ix.netcom.com.

Re: [go-nuts] Deallocate memory consumption of slice.

[robert engels]

But why would you want to allocate a new array? This would only be required if 
the MongoDB call were async, and the array cannot be reused.

If you keep allocating new arrays you are putting needless pressure on the GC.

> On Aug 26, 2024, at 11:18 AM, Marvin Renich  wrote:
> 
> * Gowtham Raj mailto:rgowtham...@gmail.com>> [240826 
> 11:31]:
>> 
>> 
>> I have a sample go snippet which pulls 50k rows from source DB and inserts 
>> into target DB. https://go.dev/play/p/IVANZyVUbkm
>> 
>> While doing so I create a empty slice and add operations to it. When the 
>> length of slice is > 10K, I do a bulk write to DB.
>> 
>> The problem here is this script eats more memory as it runs, I suspect 
>> issue is with the slice which gets appended periodically. I try to do a 
>> copy as suggested in few articles and reset it but it never gives up on the 
>> memory and the script consumes more and more as it runs.
>> 
>> Any idea as to who to release the memory of slice every 10k docs ?
>> 
>> I have reset the slice, but memory is not returned back.
> 
> Copying the slice to modelsCopy does nothing but allocate more memory,
> which will be freed when the garbage collector gets around to it.  Take
> out this code.
> 
> The statement
> 
>models = models[:0]
> 
> does not free the memory allocated for the slice's backing array, but it
> does cause future appends to reuse slice elements from the beginning of
> the array.  What you probably want is
> 
>models = []mongo.WriteModel{}
> 
> which will mark the old backing array for collection and create a new
> slice.
> 
> ...Marvin
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/Zsyqz8zgxvf8cjYE%40basil.wdw 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/BFC0E448-0FBF-438E-807B-3874C9ED18E6%40ix.netcom.com.

Re: [go-nuts] Slice of interfaces taking up so much memory

[robert engels]

Actually, this is interesting, since it requires the new capacity to be less 
than the existing capacity, I surmise it shares the underlying array, but since 
the new capacity is 0, as soon as you append it is going to allocate a new 
array - which you don’t need or want.

> On Aug 26, 2024, at 10:48 AM, robert engels  wrote:
> 
> actually, change it to models = models[0:0], not models=models[0:0:0], 
> otherwise you are causing new arrays to be allocated on each loop.
> 
>> On Aug 26, 2024, at 10:45 AM, robert engels > <mailto:reng...@ix.netcom.com>> wrote:
>> 
>> It is working as expected. You allocate models to hold 10k elements - you 
>> never put more than 10k elements in it, so it will not consume additional 
>> memory. (You should probably make the comparison == 1, not > 1 to 
>> avoid the extra array allocation.
>> 
>> GC is non deterministic - so it will only free memory if/when it needs to. I 
>> would perform a heap dump to understand exactly what is consuming the memory 
>> if it is continually increasing - I suspect it may be in the MongoDB driver 
>> - or it is just dead objects waiting to be collected.
>> 
>>> On Aug 26, 2024, at 8:24 AM, Gowtham Raj >> <mailto:rgowtham...@gmail.com>> wrote:
>>> 
>>> Hello team,
>>> 
>>> MongoDB Go bulk write takes up an slice of interfaces which is slice of 
>>> operations.
>>> https://go.dev/play/p/2vcHx5EFK5x <https://go.dev/play/p/2vcHx5EFK5x> 
>>> While doing so I create a empty slice and add operations to it. When the 
>>> length of slice is > 10K, I do a bulk write to DB.
>>> 
>>> The problem here is this script eats more memory as it runs, I suspect 
>>> issue is with the slice which gets appended periodically. I try to reset 
>>> the slice but it never gets freed up. Alloc in runtime.MemStats grows to 1 
>>> GB immediately within 1 minute.
>>> 
>>> Any idea as to who to release the memory of slice every 10k docs ?
>>> The length and cap of slice goes back to zero, but GC does not clean it up. 
>>> Not sure if since its in the for loop the variable is still being 
>>> referenced.
>>> How do solve this problem ?
>>> Regards,
>>> Gowtham
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to golang-nuts+unsubscr...@googlegroups.com 
>>> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/0be9320a-e004-49b0-a88f-79158c1ed0d2n%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/golang-nuts/0be9320a-e004-49b0-a88f-79158c1ed0d2n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EAE7F3B6-F593-45A0-89F9-8B0C7329269A%40ix.netcom.com.

Re: [go-nuts] Slice of interfaces taking up so much memory

[robert engels]

actually, change it to models = models[0:0], not models=models[0:0:0], 
otherwise you are causing new arrays to be allocated on each loop.

> On Aug 26, 2024, at 10:45 AM, robert engels  wrote:
> 
> It is working as expected. You allocate models to hold 10k elements - you 
> never put more than 10k elements in it, so it will not consume additional 
> memory. (You should probably make the comparison == 1, not > 1 to 
> avoid the extra array allocation.
> 
> GC is non deterministic - so it will only free memory if/when it needs to. I 
> would perform a heap dump to understand exactly what is consuming the memory 
> if it is continually increasing - I suspect it may be in the MongoDB driver - 
> or it is just dead objects waiting to be collected.
> 
>> On Aug 26, 2024, at 8:24 AM, Gowtham Raj > <mailto:rgowtham...@gmail.com>> wrote:
>> 
>> Hello team,
>> 
>> MongoDB Go bulk write takes up an slice of interfaces which is slice of 
>> operations.
>> https://go.dev/play/p/2vcHx5EFK5x <https://go.dev/play/p/2vcHx5EFK5x> 
>> While doing so I create a empty slice and add operations to it. When the 
>> length of slice is > 10K, I do a bulk write to DB.
>> 
>> The problem here is this script eats more memory as it runs, I suspect issue 
>> is with the slice which gets appended periodically. I try to reset the slice 
>> but it never gets freed up. Alloc in runtime.MemStats grows to 1 GB 
>> immediately within 1 minute.
>> 
>> Any idea as to who to release the memory of slice every 10k docs ?
>> The length and cap of slice goes back to zero, but GC does not clean it up. 
>> Not sure if since its in the for loop the variable is still being referenced.
>> How do solve this problem ?
>> Regards,
>> Gowtham
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0be9320a-e004-49b0-a88f-79158c1ed0d2n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/0be9320a-e004-49b0-a88f-79158c1ed0d2n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F521F794-A29B-4807-99C6-787EF5163A40%40ix.netcom.com.

Re: [go-nuts] Slice of interfaces taking up so much memory

[robert engels]

It is working as expected. You allocate models to hold 10k elements - you never 
put more than 10k elements in it, so it will not consume additional memory. 
(You should probably make the comparison == 1, not > 1 to avoid the 
extra array allocation.

GC is non deterministic - so it will only free memory if/when it needs to. I 
would perform a heap dump to understand exactly what is consuming the memory if 
it is continually increasing - I suspect it may be in the MongoDB driver - or 
it is just dead objects waiting to be collected.

> On Aug 26, 2024, at 8:24 AM, Gowtham Raj  wrote:
> 
> Hello team,
> 
> MongoDB Go bulk write takes up an slice of interfaces which is slice of 
> operations.
> https://go.dev/play/p/2vcHx5EFK5x 
> While doing so I create a empty slice and add operations to it. When the 
> length of slice is > 10K, I do a bulk write to DB.
> 
> The problem here is this script eats more memory as it runs, I suspect issue 
> is with the slice which gets appended periodically. I try to reset the slice 
> but it never gets freed up. Alloc in runtime.MemStats grows to 1 GB 
> immediately within 1 minute.
> 
> Any idea as to who to release the memory of slice every 10k docs ?
> The length and cap of slice goes back to zero, but GC does not clean it up. 
> Not sure if since its in the for loop the variable is still being referenced.
> How do solve this problem ?
> Regards,
> Gowtham
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/0be9320a-e004-49b0-a88f-79158c1ed0d2n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/281D4ADC-D62F-4EEB-84CF-62D143E3F8BC%40ix.netcom.com.

Re: [go-nuts] gc: optimize JMP to RET instructions

[robert engels]

My understanding is that optimizations like this are almost never worth it on 
modern processors - the increased code size works against the modern branch 
predictor and speculative executions - vs the single shared piece of code - 
there is less possibilities and thus instructions to preload.

> On Aug 14, 2024, at 11:46 AM, Arseny Samoylov  
> wrote:
> 
> > Won’t the speculative/parallel execution by most processors make the JMP 
> > essentially a no-op?
> I guess you are right, but this is true when JMP destination already in 
> instruction buffer. I guess most of these cases are when JMP leads to RET 
> inside on function, so indeed this optimization will have almost zero effect. 
> But if RET instruction appears to be far enough, I guess this optimization 
> can be meaningful.
> 
> On Wednesday 14 August 2024 at 19:40:22 UTC+3 robert engels wrote:
> Won’t the speculative/parallel execution by most processors make the JMP 
> essentially a no-op?
> 
> See 
> https://stackoverflow.com/questions/5127833/meaningful-cost-of-the-jump-instruction
>  
> <https://stackoverflow.com/questions/5127833/meaningful-cost-of-the-jump-instruction>
> 
> 
>> On Aug 14, 2024, at 11:31 AM, Arseny Samoylov > > wrote:
>> 
> 
>> Thank you for your answer!
>> 
>> > We generally don't do optimizations like that directly on assembly.
>> I definitely agree. But this is also a pattern for generated code.
>> 
>> > and concerns about debuggability (can you set a breakpoint on each return 
>> > in the source?) also matter
>> This is an interesting problem that I haven't thought about, thank you!
>> 
>> > That is a JMP to the LDP instruction, not directly to the RET.
>> Yes, but on Prog representation it is. I mentioned it when pointed out 
>> problem with increasing code size (RET translates to multiple instructions).
>> 
>> >  There's some discussion here https://github.com/golang/go/issues/24936 
>> > <https://github.com/golang/go/issues/24936>
>> I am grateful for the link to the discussion. In this discussion, you 
>> mentioned yours abandoned CL  
>> <https://github.com/golang/go/issues/24936#issuecomment-383253003>that 
>> actually does the contrary of my optimization =).
>> 
>> >  It would need benchmarks demonstrating it is worth it
>> Can you please provide some suggestions for benchmarks? I tried bent, but I 
>> would like to test on some other benchmarks. 
>> 
>> Thank you in advance!
>> On Wednesday 14 August 2024 at 03:59:55 UTC+3 Keith Randall wrote:
>> We generally don't do optimizations like that directly on assembly. In fact, 
>> we used to do some like that but they have been removed.
>> We want the generated machine code to faithfully mirror the assembly input. 
>> People writing assembly have all kind of reasons for laying out instructions 
>> in particular ways (better for various caches, etc) that we don't want to 
>> disrupt.
>> 
>> If the Go compiler is generating such a pattern, we can optimize that. 
>> There's some discussion here https://github.com/golang/go/issues/24936 
>> <https://github.com/golang/go/issues/24936> but nothing substantive came of 
>> it. It would need benchmarks demonstrating it is worth it, and concerns 
>> about debuggability (can you set a breakpoint on each return in the source?) 
>> also matter.
>> 
>> > Ps: example of JMP to RET from runtime:
>> 
>> That is a JMP to the LDP instruction, not directly to the RET.
>> On Tuesday, August 13, 2024 at 10:10:58 AM UTC-7 Arseny Samoylov wrote:
>> Hello community, recently I found that gc generates a lot of JMP to RET 
>> instructions and there is no optimization for that. Consider this example:
>> 
>> ```
>> // asm_arm64.s
>> #include "textflag.h"
>>  
>> TEXT ·jmp_to_ret(SB), NOSPLIT, $0-0
>> JMP ret
>> ret:
>> RET
>> ```
>> This compiles to :
>> ```
>> TEXT main.jmp_to_ret.abi0(SB) asm_arm64.s
>>   asm_arm64.s:4 0x77530 1401JMP 
>> 1(PC)
>>   asm_arm64.s:6 0x77534 d65f03c0RET
>> ```
>> 
>> Obviously, it can be optimized just to RET instruction.
>> So I made a patch that replaces JMP to RET with RET instruction (on Prog 
>> representation):
>> ```
>> diff --git a/src/cmd/internal/obj/pass.go b/src/cmd/internal/obj/pass.go
>> index 066b779539..87f1121641 100644
>> --- a/src/cmd/internal/obj/pass.go
>> +++ b/src/cmd/internal/o

Re: [go-nuts] gc: optimize JMP to RET instructions

[robert engels]

Won’t the speculative/parallel execution by most processors make the JMP 
essentially a no-op?

See 
https://stackoverflow.com/questions/5127833/meaningful-cost-of-the-jump-instruction

> On Aug 14, 2024, at 11:31 AM, Arseny Samoylov  
> wrote:
> 
> Thank you for your answer!
> 
> > We generally don't do optimizations like that directly on assembly.
> I definitely agree. But this is also a pattern for generated code.
> 
> > and concerns about debuggability (can you set a breakpoint on each return 
> > in the source?) also matter
> This is an interesting problem that I haven't thought about, thank you!
> 
> > That is a JMP to the LDP instruction, not directly to the RET.
> Yes, but on Prog representation it is. I mentioned it when pointed out 
> problem with increasing code size (RET translates to multiple instructions).
> 
> >  There's some discussion here https://github.com/golang/go/issues/24936 
> > 
> I am grateful for the link to the discussion. In this discussion, you 
> mentioned yours abandoned CL  
> that 
> actually does the contrary of my optimization =).
> 
> >  It would need benchmarks demonstrating it is worth it
> Can you please provide some suggestions for benchmarks? I tried bent, but I 
> would like to test on some other benchmarks. 
> 
> Thank you in advance!
> On Wednesday 14 August 2024 at 03:59:55 UTC+3 Keith Randall wrote:
> We generally don't do optimizations like that directly on assembly. In fact, 
> we used to do some like that but they have been removed.
> We want the generated machine code to faithfully mirror the assembly input. 
> People writing assembly have all kind of reasons for laying out instructions 
> in particular ways (better for various caches, etc) that we don't want to 
> disrupt.
> 
> If the Go compiler is generating such a pattern, we can optimize that. 
> There's some discussion here https://github.com/golang/go/issues/24936 
>  but nothing substantive came of 
> it. It would need benchmarks demonstrating it is worth it, and concerns about 
> debuggability (can you set a breakpoint on each return in the source?) also 
> matter.
> 
> > Ps: example of JMP to RET from runtime:
> 
> That is a JMP to the LDP instruction, not directly to the RET.
> On Tuesday, August 13, 2024 at 10:10:58 AM UTC-7 Arseny Samoylov wrote:
> Hello community, recently I found that gc generates a lot of JMP to RET 
> instructions and there is no optimization for that. Consider this example:
> 
> ```
> // asm_arm64.s
> #include "textflag.h"
>  
> TEXT ·jmp_to_ret(SB), NOSPLIT, $0-0
> JMP ret
> ret:
> RET
> ```
> This compiles to :
> ```
> TEXT main.jmp_to_ret.abi0(SB) asm_arm64.s
>   asm_arm64.s:4 0x77530 1401JMP 
> 1(PC)
>   asm_arm64.s:6 0x77534 d65f03c0RET
> ```
> 
> Obviously, it can be optimized just to RET instruction.
> So I made a patch that replaces JMP to RET with RET instruction (on Prog 
> representation):
> ```
> diff --git a/src/cmd/internal/obj/pass.go b/src/cmd/internal/obj/pass.go
> index 066b779539..87f1121641 100644
> --- a/src/cmd/internal/obj/pass.go
> +++ b/src/cmd/internal/obj/pass.go
> @@ -174,8 +174,16 @@ func linkpatch(ctxt *Link, sym *LSym, newprog ProgAlloc) 
> {
> continue
> }
> p.To.SetTarget(brloop(p.To.Target()))
> -   if p.To.Target() != nil && p.To.Type == TYPE_BRANCH {
> -   p.To.Offset = p.To.Target().Pc
> +   if p.To.Target() != nil {
> +   if p.As == AJMP && p.To.Target().As == ARET {
> +   p.As = ARET
> +   p.To = p.To.Target().To
> +   continue
> +   }
> +
> +   if p.To.Type == TYPE_BRANCH {
> +   p.To.Offset = p.To.Target().Pc
> +   }
> }
> }
>  }
> ```
> You can find this patch on my GH 
> .
> 
> I encountered few problems:
> * Increase in code size - because RET instruction can translate in multiple 
> instructions (ldp, add, and ret - on arm64 for example):
> .text section of simple go program that calls function from above increases 
> in 0x3D0 bytes; go binary itself increases in 0x2570 (almost 10KB) in .text 
> section size 
> (this is for arm64 binaries)
> * Optimization on Prog representation is too late, and example above 
> translates to:
> ```
> TEXT main.jmp_to_ret.abi0(SB) asm_arm64.s
>   asm_arm64.s:4 0x77900 d65f03c0RET
>   asm_arm64.s:6 0x77904 d65f03c0RET
> ```
> (no dead code elimination was done =( )
> 
> So I am looking for some ideas.

Re: [go-nuts] GOMAXPROCS?

[robert engels]

Thanks.

> On Aug 12, 2024, at 1:13 PM, Ian Lance Taylor  wrote:
> 
> On Mon, Aug 12, 2024 at 6:17 AM Robert Engels  wrote:
>> 
>> Revisiting an old thread 
>> https://groups.google.com/g/golang-nuts/c/jPb_h3TvlKE/m/qdoHhxXeAwAJ
>> 
>> Is it still the case that high disk IO throughput requires a GOMAXPROCS 
>> seething higher than the number of cpus on the system?
> 
> It's true that we don't use the poller for local file operations on
> most systems.  So if your program is overloading the file system, it
> is possible for goroutines to be delayed waiting for file I/O to
> finish.  Those delays won't be for long, as the scheduler will step in
> and start new threads as needed.  However, it won't do that right
> away; I think the delay can be up to 10ms.  In such a case it can be
> useful to increase GOMAXPROCS to permit queuing up more file I/O more
> quickly, or to do other non-I/O related activity.
> 
>> Are there any other cases where this might be required?
> 
> Not that I'm aware of.
> 
> Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0E7A8652-6D15-48C1-ACFD-678A62798C92%40ix.netcom.com.

[go-nuts] GOMAXPROCS?

[Robert Engels]

Hi,

Revisiting an old thread 
https://groups.google.com/g/golang-nuts/c/jPb_h3TvlKE/m/qdoHhxXeAwAJ

Is it still the case that high disk IO throughput requires a GOMAXPROCS 
seething higher than the number of cpus on the system?

Are there any other cases where this might be required?

Thanks. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0C98358D-A3FB-412C-BCB7-2C348F4A58BD%40ix.netcom.com.

Re: [go-nuts] Order of calls to conn.Read and conn.Write on server and client

[Robert Engels]

Also the proper use of Go routines allow you to implement async without the messiness of callbacks. On Aug 11, 2024, at 1:49 PM, Robert Engels  wrote:This is not true. The synchronous requirements are based on higher level protocol requirements. You can just as easily do async with Reader and Writer. Eg. The client can send multiple requests and the server could respond to these out of order (I.e the response contains the request id) or not respond at all. You might want to look at using gRPC which will help you understand the model better. On Aug 11, 2024, at 1:44 PM, 'lijh8' via golang-nuts  wrote:Hi community,In my little example:if server reads first, then writes;client must writes first, then read;then client must performs the operations in reverse order as server,or both server and client will block.In C++ boost.asio, the order of calls to async_read, async_write does not matter in server and client.Is there this kind of ` async ` read and write ability in golang packages?Thanks```// server.go:func main() {	logfile := "logfile.log"	logfd, err := log2.InitLog(logfile)	if err == nil {		defer logfd.Close()	}	if len(os.Args) != 2 {		log.Printf("Usage: ./server ")		os.Exit(-1)	}	port := os.Args[1]	ln, err := net.Listen("tcp", ":"+port)	if err != nil {		log.Println(err)		os.Exit(-1)	}	for {		conn, err := ln.Accept()		if err != nil {			log.Println(err)			os.Exit(-1)		}		go handleConnection(conn)	}}func handleConnection(conn net.Conn) {	defer conn.Close()	msgID := 0	buffer := make([]byte, 1024)	for {		n, err := conn.Read(buffer)		if err != nil {			log.Println(err, n)			break		}		fmt.Println(string(buffer))		msgID++		msg := fmt.Sprintf("msg from server with msgID: %d", msgID)		n, err = conn.Write([]byte(msg))		if err != nil {			log.Println(err, n)			break		}	}}// client.go:func main() {	logfile := "logfile.log"	logfd, err := log2.InitLog(logfile)	if err == nil {		defer logfd.Close()	}	if len(os.Args) != 4 {		log.Printf("Usage: ./client   ")		os.Exit(-1)	}	serverIP := os.Args[1]	port := os.Args[2]	tag := os.Args[3]	conn, err := net.Dial("tcp", serverIP+":"+port)	if err != nil {		log.Println(err)		os.Exit(-1)	}	handleConnection(conn, tag)}func handleConnection(conn net.Conn, tag string) {	defer conn.Close()	msgID := 0	buffer := make([]byte, 1024)	for {		msgID++		msg := fmt.Sprintf("msg from client: %s, msgID: %d", tag, msgID)		n, err := conn.Write([]byte(msg))		if err != nil {			log.Println(err, n)			break		}		n, err = conn.Read(buffer)		if err != nil {			log.Println(err, n)			break		}		fmt.Println(string(buffer))	}}```



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/tencent_16864CF32F826D31908D052707F2E2621709%40qq.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA2745A0-8C46-4037-A8EF-04A4DDC29BB9%40ix.netcom.com.

Re: [go-nuts] Order of calls to conn.Read and conn.Write on server and client

[Robert Engels]

This is not true. The synchronous requirements are based on higher level protocol requirements. You can just as easily do async with Reader and Writer. Eg. The client can send multiple requests and the server could respond to these out of order (I.e the response contains the request id) or not respond at all. You might want to look at using gRPC which will help you understand the model better. On Aug 11, 2024, at 1:44 PM, 'lijh8' via golang-nuts  wrote:Hi community,In my little example:if server reads first, then writes;client must writes first, then read;then client must performs the operations in reverse order as server,or both server and client will block.In C++ boost.asio, the order of calls to async_read, async_write does not matter in server and client.Is there this kind of ` async ` read and write ability in golang packages?Thanks```// server.go:func main() {	logfile := "logfile.log"	logfd, err := log2.InitLog(logfile)	if err == nil {		defer logfd.Close()	}	if len(os.Args) != 2 {		log.Printf("Usage: ./server ")		os.Exit(-1)	}	port := os.Args[1]	ln, err := net.Listen("tcp", ":"+port)	if err != nil {		log.Println(err)		os.Exit(-1)	}	for {		conn, err := ln.Accept()		if err != nil {			log.Println(err)			os.Exit(-1)		}		go handleConnection(conn)	}}func handleConnection(conn net.Conn) {	defer conn.Close()	msgID := 0	buffer := make([]byte, 1024)	for {		n, err := conn.Read(buffer)		if err != nil {			log.Println(err, n)			break		}		fmt.Println(string(buffer))		msgID++		msg := fmt.Sprintf("msg from server with msgID: %d", msgID)		n, err = conn.Write([]byte(msg))		if err != nil {			log.Println(err, n)			break		}	}}// client.go:func main() {	logfile := "logfile.log"	logfd, err := log2.InitLog(logfile)	if err == nil {		defer logfd.Close()	}	if len(os.Args) != 4 {		log.Printf("Usage: ./client   ")		os.Exit(-1)	}	serverIP := os.Args[1]	port := os.Args[2]	tag := os.Args[3]	conn, err := net.Dial("tcp", serverIP+":"+port)	if err != nil {		log.Println(err)		os.Exit(-1)	}	handleConnection(conn, tag)}func handleConnection(conn net.Conn, tag string) {	defer conn.Close()	msgID := 0	buffer := make([]byte, 1024)	for {		msgID++		msg := fmt.Sprintf("msg from client: %s, msgID: %d", tag, msgID)		n, err := conn.Write([]byte(msg))		if err != nil {			log.Println(err, n)			break		}		n, err = conn.Read(buffer)		if err != nil {			log.Println(err, n)			break		}		fmt.Println(string(buffer))	}}```



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/tencent_16864CF32F826D31908D052707F2E2621709%40qq.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/678FE9AE-3AD5-436E-831E-ECF7810ABB80%40ix.netcom.com.

Re: [go-nuts] Why do I get "The connection was reset" error when uploading a ~10MB file, and how to avoid reading the file into memory?

[Robert Engels]

Check out this https://jesseduffield.com/Golang-IO-Cookbook/ to see how to transfer the body to a file without reading the whole thing into memory. On Aug 2, 2024, at 12:41 PM, Robert Engels  wrote:You most likely have to read the incoming data. Depends on the framework. The tcp connection is reused so if you don’t read all of the sent data then the next request will be corrupted. I am not sure why it works with small files - I guess that the framework is reading some of the data into the request body buffer - and if it reads the full request body in doing so that would explain it. On Aug 2, 2024, at 12:35 PM, Denis Korovin  wrote:Hi everyone,I'm running into an issue with my Go server when trying to upload files around 10MB using a simple file upload form. In my browser, I'm seeing a "The connection was reset" error, and I'm not sure what is causing this problem or how to fix it.Here’s my current setup:  https://go.dev/play/p/7TVn-njfeiHWhen I attempt to upload a file around 10MB through this form, the browser gives me a "The connection was reset" error. I’ve verified that smaller files upload without any issues.My uploadHandler currently does nothing but send a 200 OK status. Ideally, I'd like to process the file upload without reading it entirely into memory, to handle larger file uploads efficiently.Can anyone help me understand why I'm getting this error and how to properly handle large file uploads without reading them completely into memory? Any guidance or examples would be greatly appreciated.Thanks in advance!



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f9cbb87f-c597-4249-a13e-fbc634b38e1bn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/FAC045FC-BB12-4255-B42E-858CD9C154BA%40ix.netcom.com.

Re: [go-nuts] Why do I get "The connection was reset" error when uploading a ~10MB file, and how to avoid reading the file into memory?

[Robert Engels]

You most likely have to read the incoming data. Depends on the framework. The tcp connection is reused so if you don’t read all of the sent data then the next request will be corrupted. I am not sure why it works with small files - I guess that the framework is reading some of the data into the request body buffer - and if it reads the full request body in doing so that would explain it. On Aug 2, 2024, at 12:35 PM, Denis Korovin  wrote:Hi everyone,I'm running into an issue with my Go server when trying to upload files around 10MB using a simple file upload form. In my browser, I'm seeing a "The connection was reset" error, and I'm not sure what is causing this problem or how to fix it.Here’s my current setup:  https://go.dev/play/p/7TVn-njfeiHWhen I attempt to upload a file around 10MB through this form, the browser gives me a "The connection was reset" error. I’ve verified that smaller files upload without any issues.My uploadHandler currently does nothing but send a 200 OK status. Ideally, I'd like to process the file upload without reading it entirely into memory, to handle larger file uploads efficiently.Can anyone help me understand why I'm getting this error and how to properly handle large file uploads without reading them completely into memory? Any guidance or examples would be greatly appreciated.Thanks in advance!



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f9cbb87f-c597-4249-a13e-fbc634b38e1bn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/22E935CC-E806-498F-8ECB-4487A6783965%40ix.netcom.com.

Re: [go-nuts] photoslibrary/v1 no longer available

[robert engels]

There is also a third-party library that might work or be a great starting 
point: https://github.com/gphotosuploader/google-photos-api-client-go

> On Aug 1, 2024, at 9:15 AM, robert engels  wrote:
> 
> It should be fairly straightforward to use the Rest api directly 
> https://developers.google.com/photos/library/guides/get-started 
> <https://developers.google.com/photos/library/guides/get-started>
> 
> You can also look at the Java or PhP client library and reimplement in Go - 
> sadly it doesn’t seem they provide the raw proto files you could use to 
> generate the Go structs so you’ll have to do it by hand for the operations 
> you need.
> 
>> On Aug 1, 2024, at 1:41 AM, Pratik Tamgole > <mailto:pratiktamg...@gmail.com>> wrote:
>> 
>> Hi,
>> Thank you for all the information.
>> 
>> Have tried using v0.3.2 and 0.189.0, but still getting the broken import 
>> error, the file does not seem to be there.
>> 
>> Also James, tried the 
>> https://pkg.go.dev/github.com/mathieubrun/go-google-photos-api/photoslibrary/v1
>>  
>> <https://pkg.go.dev/github.com/mathieubrun/go-google-photos-api/photoslibrary/v1>
>>  but to no avail.
>> Changing the webapp on my side to be able to upload straight from the device 
>> rather than uploading it from google drive. 
>> 
>> Can you recommend resources to me if I want to make a repo like 
>> "https://pkg.go.dev/github.com/mathieubrun/go-google-photos-api/photoslibrary/v1
>>  
>> <https://pkg.go.dev/github.com/mathieubrun/go-google-photos-api/photoslibrary/v1>"
>>  as a substitute?
>>  
>> Again thanks for all your help!
>> 
>> Regards,
>> Pratik Tamgole.
>> 
>> On Wednesday, July 31, 2024 at 7:30:37 PM UTC+5:30 peterGo wrote:
>> Pratik Tamgole
>> 
>> Changes
>> https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md 
>> <https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md>
>> 
>> v0.4.0
>> 
>> Deletes photoslibrary/v1. The photoslibrary team hopes to fully support Go 
>> in the near future, but this autogenerated library is ready to be sunset. If 
>> you rely on this client, please vendor this library at v0.3.2.
>> 
>> For v0.3.2:
>> https://github.com/googleapis/google-api-go-client/tree/v0.3.2/photoslibrary/v1
>>  
>> <https://github.com/googleapis/google-api-go-client/tree/v0.3.2/photoslibrary/v1>
>> 
>> peter
>> 
>> On Wednesday, July 31, 2024 at 9:05:10 AM UTC-4 Pratik Tamgole wrote:
>> Hi,
>> 
>> I'm trying to use the Go library photoslibrary/v1 but getting an error 
>> message saying:
>>  
>> "google.golang.org/api/photoslibrary/v1 
>> <http://google.golang.org/api/photoslibrary/v1>: module 
>> google.golang.org/api@latest <http://google.golang.org/api@latest> found 
>> (v0.189.0), but does not contain package 
>> google.golang.org/api/photoslibrary/v1 
>> <http://google.golang.org/api/photoslibrary/v1>"
>> 
>> And it seems they have removed it from the source.
>> 
>> How do I go about replacing this? Any other library that can help my app 
>> integrate with the GooglePhotos app? And if there is any other solution to 
>> working with the photoslibrary library? 
>> 
>> 
>> Regards, 
>> Pratik Tamgole
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/ebdd26be-462f-4924-8085-c0a9f8f9cd70n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/ebdd26be-462f-4924-8085-c0a9f8f9cd70n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/10081CDD-3BDA-4362-92FE-5398F4BBE3C7%40ix.netcom.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/10081CDD-3BDA-4362-92FE-5398F4BBE3C7%40ix.netcom.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/F50EC8E8-D246-48E7-ACE9-AA8DF92AA109%40ix.netcom.com.

Re: [go-nuts] photoslibrary/v1 no longer available

[robert engels]

It should be fairly straightforward to use the Rest api directly 
https://developers.google.com/photos/library/guides/get-started 


You can also look at the Java or PhP client library and reimplement in Go - 
sadly it doesn’t seem they provide the raw proto files you could use to 
generate the Go structs so you’ll have to do it by hand for the operations you 
need.

> On Aug 1, 2024, at 1:41 AM, Pratik Tamgole  wrote:
> 
> Hi,
> Thank you for all the information.
> 
> Have tried using v0.3.2 and 0.189.0, but still getting the broken import 
> error, the file does not seem to be there.
> 
> Also James, tried the 
> https://pkg.go.dev/github.com/mathieubrun/go-google-photos-api/photoslibrary/v1
>  
> 
>  but to no avail.
> Changing the webapp on my side to be able to upload straight from the device 
> rather than uploading it from google drive. 
> 
> Can you recommend resources to me if I want to make a repo like 
> "https://pkg.go.dev/github.com/mathieubrun/go-google-photos-api/photoslibrary/v1
>  
> "
>  as a substitute?
>  
> Again thanks for all your help!
> 
> Regards,
> Pratik Tamgole.
> 
> On Wednesday, July 31, 2024 at 7:30:37 PM UTC+5:30 peterGo wrote:
> Pratik Tamgole
> 
> Changes
> https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md 
> 
> 
> v0.4.0
> 
> Deletes photoslibrary/v1. The photoslibrary team hopes to fully support Go in 
> the near future, but this autogenerated library is ready to be sunset. If you 
> rely on this client, please vendor this library at v0.3.2.
> 
> For v0.3.2:
> https://github.com/googleapis/google-api-go-client/tree/v0.3.2/photoslibrary/v1
>  
> 
> 
> peter
> 
> On Wednesday, July 31, 2024 at 9:05:10 AM UTC-4 Pratik Tamgole wrote:
> Hi,
> 
> I'm trying to use the Go library photoslibrary/v1 but getting an error 
> message saying:
>  
> "google.golang.org/api/photoslibrary/v1 
> : module 
> google.golang.org/api@latest  found 
> (v0.189.0), but does not contain package 
> google.golang.org/api/photoslibrary/v1 
> "
> 
> And it seems they have removed it from the source.
> 
> How do I go about replacing this? Any other library that can help my app 
> integrate with the GooglePhotos app? And if there is any other solution to 
> working with the photoslibrary library? 
> 
> 
> Regards, 
> Pratik Tamgole
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/ebdd26be-462f-4924-8085-c0a9f8f9cd70n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/10081CDD-3BDA-4362-92FE-5398F4BBE3C7%40ix.netcom.com.

Re: [go-nuts] How to refresh database/sql MySQL connection username and password

[Robert Engels]

I am fairly certain the reconnection details, etc will be driver specific.  Many dbs will not drop the current connections when changing passwords - only used when a new connection is created. The easiest solution is to set the max connection lifetime on the pool, and change the password there to match. How you coordinate that will be more involved. Might be easier to detect the invalid password error and reset the pool. On Jul 29, 2024, at 7:32 AM, prasa...@gmail.com  wrote:I have a long running daemon written in Go, that listens to a port and spins up multiple go routines for every new connection to handle the data. There is a global variable context that has db Conn(*sql.DB) that is assigned connection context returned by database/sql library’s open() function during initialization time.for security reasons we want to rotate db username & password periodically. I'm looking for best possible way to achieve same but i'm not able to find any pointers to it in documentation. Can anyone please suggest methods to handle the same?I'm also not able find info about auto reconnecting details in case of DB failure. Just wondering how long driver will keep retyring in case of DB shutdown or failure?closest one that i got is:  https://stackoverflow.com/questions/61921525/how-to-update-db-connection-in-gorm-with-new-aws-token/77287404#77287404.  but its good for only postgress.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/db61ab36-ed98-4b86-bdd3-7125fe85e284n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/09ABA647-58EA-4F33-923B-8AA2461FFDE8%40ix.netcom.com.

Re: [go-nuts] Re: how to start with go?

[Robert Engels]

A few people have reached out to tell me they found github.com/robaho/go-trader worthwhile when learning. It was a project I initially created to learn Go myself. It touches on all of the major food groups except db access. On Jul 23, 2024, at 9:01 AM, Guilherme de Oliveira Amorim  wrote:Well, usually, when I want to learn a new programming language, I create a project using the architecture I'm most familiar with, which in this case is Clean Architecture.This way, basic elements of the language are studied in an intuitive and contextualized way. This helps to consolidate the learning and gives a clearer view of what the language is capable of and what its limitations are.It works well for me!Hope this helps!Em terça-feira, 23 de julho de 2024 às 09:33:14 UTC-3, Kowsik Vunnam escreveu:Hi everyone I am new to coding and I learned Golang. I dont know how to start contributing or how to search for issues or how to identify issues, My question is how do people identify the issues how do they test the code?



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c05612e6-6684-4345-8ec3-161d84bcb895n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7F914CA8-39E1-4EBB-938D-154BC9B6F859%40ix.netcom.com.

Re: [go-nuts] Mutex profiling question

[Robert Engels]

Thank you for the very detailed explanation. What I am struggling with is that even if the routines do a little work and go back to sleep (which is probably the case here) - as I increase the number of external clients I would expect there is always a routine ready to run. What I am observing is that the throughput is capped. Eg with 10 clients each processes 3k requests a second, but with 20 clients each processes 1.5k requests a sec. Yet, the cpu usage is about the same and the IO rates about the same. Which points to scheduling delays - but I’ll confess after reading your response I still am not sure how to diagnose (what other tools I can look at - the trace seems too noisy). Is there some way to get high level scheduling statistics?On Jul 18, 2024, at 6:31 PM, Rhys Hiltner  wrote:Hi Robert,First, note that the contention profile for runtime-internal locks doesn't correctly blame the part of the code that _caused_ delay: with Go 1.22 and 1.23, the call stacks are of (runtime-internal) lock users that _experienced_ delay. That's https://go.dev/issue/66999, and those odd semantics are the reason that those stacks aren't shown by default (and instead require the GODEBUG). That was briefly fixed in late May, but caused some slowdowns and so couldn't be part of the Go 1.23 release.Contention within findRunnable is probably on sched.lock, which protects the global portion of the scheduler. (Often, scheduling only needs to involve each P's local run queue.)> Is this a side effect of the scheduler, and it is really blocked in the unlock call waiting for another routine to be runnable?There really is a thread (an M with a P) that is blocked. It's not waiting for a G to be runnable .. depending on the exact part of findRunnable, it might be waiting to see whether a goroutine is runnable or not, or the M might be trying to go to sleep because it's determined that there's no work. The profile also shows runtime.goschedImpl, which means there's a G calling runtime.Gosched, and its M (and P) are blocked trying to add the G to the global run queue.As for what's causing it ... consider how much work the program's goroutines do when they run (maybe "only tens of microseconds"), how and how often the program uses Gosched (maybe "a bit too much", though I can't quantify that), the setting for GOMAXPROCS (maybe "lots", though GOOS=darwin implies it won't be huge). Summing the runtime/metrics buckets for "/sched/latencies:seconds" and multiplying by 8 (runtime.gTrackingPeriod) will give an estimate of how many times the scheduler started running a goroutine ... or a runtime/trace can show all of those details at once.RhysOn Thursday, July 18, 2024 at 1:00:15 PM UTC-7 robert engels wrote:Anyone have any idea on the high internal lock time on findRunnable?On Jul 18, 2024, at 10:36 AM, robert engels <ren...@ix.netcom.com> wrote:Looks like the name was changed to runtimecontentionstacksMuch better now…(pprof) treeShowing nodes accounting for 481.75ms, 100% of 481.75ms totalDropped 39 nodes (cum <= 2.41ms)--+-      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 --+-                                          401.25ms 83.51% |   runtime.findRunnable (inline)                                           35.26ms  7.34% |   runtime.sysmon (inline)                                           17.37ms  3.62% |   runtime.stopm (inline)                                           14.01ms  2.92% |   runtime.saveBlockEventStack (inline)                                            6.07ms  1.26% |   runtime.goschedImpl (inline)                                            1.52ms  0.32% |   runtime.schedule (inline)                                            1.30ms  0.27% |   sync.(*Mutex).Unlock (inline)                                            1.30ms  0.27% |   github.com/robaho/go-trader/internal/exchange.sendMarketData (inline)                                            1.17ms  0.24% |   sync.(*Map).Swap (inline)                                            1.16ms  0.24% |   github.com/robaho/go-trader/internal/exchange.(*grpcServer).Connection (inline)  480.46ms 99.73% 99.73%   480.46ms 99.73%                | runtime.unlockbut this seems suspect, that finding the runnable should consume so much time. Is this a side effect of the scheduler, and it is really blocked in the unlock call waiting for another routine to be runnable?On Jul 18, 2024, at 10:14 AM, 'Robert Engels' via golang-nuts <golan...@googlegroups.com> wrote:I found this issue, https://github.com/golang/go/issues/57071 and tried running with GODEBUG=profileruntimelocks=1but it made no difference in the output.I am using Go 1.22.4 on OSX.Any ideas how I can d

Re: [go-nuts] this code sometimes hangs

[Robert Engels]

But I think that Done() is for the outer which prevents the broadcast. By putting it in the defer all of the subscribers should be waiting on the condition when it occurs. Caveat- not looking at code - just remembering. On Jul 18, 2024, at 3:01 PM, Lammie Jonson  wrote:>> defer goroutineRunning.Done()This doesn't work. Each invocation of subscribe() has it's own sync.WaitGroup. It's just used it appears to ensure that the enclosed goroutine is running before subscribe() returns 



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/45f26b0d-e00d-47e6-aede-ee9a70bb7d5dn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/82A1A7EE-EA99-4326-A122-7BCA889F8A32%40ix.netcom.com.

Re: [go-nuts] Mutex profiling question

['robert engels' via golang-nuts]

Anyone have any idea on the high internal lock time on findRunnable?On Jul 18, 2024, at 10:36 AM, robert engels  wrote:Looks like the name was changed to runtimecontentionstacksMuch better now…(pprof) treeShowing nodes accounting for 481.75ms, 100% of 481.75ms totalDropped 39 nodes (cum <= 2.41ms)--+-      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 --+-                                          401.25ms 83.51% |   runtime.findRunnable (inline)                                           35.26ms  7.34% |   runtime.sysmon (inline)                                           17.37ms  3.62% |   runtime.stopm (inline)                                           14.01ms  2.92% |   runtime.saveBlockEventStack (inline)                                            6.07ms  1.26% |   runtime.goschedImpl (inline)                                            1.52ms  0.32% |   runtime.schedule (inline)                                            1.30ms  0.27% |   sync.(*Mutex).Unlock (inline)                                            1.30ms  0.27% |   github.com/robaho/go-trader/internal/exchange.sendMarketData (inline)                                            1.17ms  0.24% |   sync.(*Map).Swap (inline)                                            1.16ms  0.24% |   github.com/robaho/go-trader/internal/exchange.(*grpcServer).Connection (inline)  480.46ms 99.73% 99.73%   480.46ms 99.73%                | runtime.unlockbut this seems suspect, that finding the runnable should consume so much time. Is this a side effect of the scheduler, and it is really blocked in the unlock call waiting for another routine to be runnable?On Jul 18, 2024, at 10:14 AM, 'Robert Engels' via golang-nuts <golang-nuts@googlegroups.com> wrote:I found this issue, https://github.com/golang/go/issues/57071 and tried running with GODEBUG=profileruntimelocks=1but it made no difference in the output.I am using Go 1.22.4 on OSX.Any ideas how I can determine what is causing this high contention shown below:(pprof) treeShowing nodes accounting for 4440.35ms, 100% of 4440.35ms total--+-      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 --+- 4324.17ms 97.38% 97.38%  4324.17ms 97.38%                | runtime._LostContendedRuntimeLock--+-                                          116.17ms   100% |   sync.(*Map).Swap (inline)  116.17ms  2.62%   100%   116.17ms  2.62%                | sync.(*Mutex).Unlock--+-                                          116.17ms   100% |   github.com/robaho/go-trader/internal/exchange.(*grpcServer).massquote         0     0%   100%   116.17ms  2.62%                | github.com/robaho/go-trader/internal/exchange.(*exchange).Quote                                          116.17ms   100% |   github.com/robaho/go-trader/internal/exchange.sendMarketData

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/25E80A1A-DFA8-4BA6-A49C-0F8F656DFA81%40me.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8F118071-A663-4CD8-963A-0F549C2D7E9E%40me.com.

Re: [go-nuts] Mutex profiling question

[robert engels]

Looks like the name was changed to runtimecontentionstacks

Much better now…

(pprof) tree
Showing nodes accounting for 481.75ms, 100% of 481.75ms total
Dropped 39 nodes (cum <= 2.41ms)
--+-
  flat  flat%   sum%cum   cum%   calls calls% + context 
 
--+-
  401.25ms 83.51% |   
runtime.findRunnable (inline)
   35.26ms  7.34% |   runtime.sysmon 
(inline)
   17.37ms  3.62% |   runtime.stopm 
(inline)
   14.01ms  2.92% |   
runtime.saveBlockEventStack (inline)
6.07ms  1.26% |   
runtime.goschedImpl (inline)
1.52ms  0.32% |   runtime.schedule 
(inline)
1.30ms  0.27% |   
sync.(*Mutex).Unlock (inline)
1.30ms  0.27% |   
github.com/robaho/go-trader/internal/exchange.sendMarketData (inline)
1.17ms  0.24% |   sync.(*Map).Swap 
(inline)
1.16ms  0.24% |   
github.com/robaho/go-trader/internal/exchange.(*grpcServer).Connection (inline)
  480.46ms 99.73% 99.73%   480.46ms 99.73%| runtime.unlock

but this seems suspect, that finding the runnable should consume so much time. 
Is this a side effect of the scheduler, and it is really blocked in the unlock 
call waiting for another routine to be runnable?



> On Jul 18, 2024, at 10:14 AM, 'Robert Engels' via golang-nuts 
>  wrote:
> 
> I found this issue, https://github.com/golang/go/issues/57071 
> <https://github.com/golang/go/issues/57071> and tried running with 
> 
> GODEBUG=profileruntimelocks=1
> 
> but it made no difference in the output.
> 
> I am using Go 1.22.4 on OSX.
> 
> Any ideas how I can determine what is causing this high contention shown 
> below:
> 
> (pprof) tree
> Showing nodes accounting for 4440.35ms, 100% of 4440.35ms total
> --+-
>   flat  flat%   sum%cum   cum%   calls calls% + context   
>  
> --+-
>  4324.17ms 97.38% 97.38%  4324.17ms 97.38%| 
> runtime._LostContendedRuntimeLock
> --+-
>   116.17ms   100% |   
> sync.(*Map).Swap (inline)
>   116.17ms  2.62%   100%   116.17ms  2.62%| 
> sync.(*Mutex).Unlock
> --+-
>   116.17ms   100% |   
> github.com/robaho/go-trader/internal/exchange 
> <http://github.com/robaho/go-trader/internal/exchange>.(*grpcServer).massquote
>  0 0%   100%   116.17ms  2.62%| 
> github.com/robaho/go-trader/internal/exchange 
> <http://github.com/robaho/go-trader/internal/exchange>.(*exchange).Quote
>   116.17ms   100% |   
> github.com/robaho/go-trader/internal/exchange.sendMarketData 
> <http://github.com/robaho/go-trader/internal/exchange.sendMarketData>
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/25E80A1A-DFA8-4BA6-A49C-0F8F656DFA81%40me.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/25E80A1A-DFA8-4BA6-A49C-0F8F656DFA81%40me.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0F9BD4CD-4125-4CF8-A253-CD697AAD29C6%40ix.netcom.com.

Re: [go-nuts] this code sometimes hangs

[robert engels]

To clarify, the inner Go routine calls Done() at the start, so the main can get 
to the Broadcast before the inners are waiting.

Which is what Jan’s fix will fix :) (pretty sure)

> On Jul 18, 2024, at 10:20 AM, robert engels  wrote:
> 
> Because the wait on the condition is in another Go routine, so the the 
> Broadcast occurs before all of the routines are waiting. Broadcast only wakes 
> up currently waiting routines.
> 
>> On Jul 18, 2024, at 9:01 AM, Lammie Jonson > <mailto:jrubia...@gmail.com>> wrote:
>> 
>> // I am not sure why this code sometimes deadlocks, but sometimes it doesn't.
>> // Hopefully someone will have some suggestions 
>> 
>> package main
>> 
>> // ??? sometimes deadlocks
>> 
>> import (
>> "fmt"
>> "sync"
>> "time"
>> )
>> 
>> type Button struct {
>> Clicked *sync.Cond
>> }
>> 
>> func main() {
>> button := Button{
>> Clicked: sync.NewCond(&sync.Mutex{}),
>> }
>> 
>> // running on goroutine every function that passed/registered
>> // and wait, not exit until that goroutine is confirmed to be running
>> subscribe := func(c *sync.Cond, param string, fn func(s string)) {
>> var goroutineRunning sync.WaitGroup
>> var cnt = 0
>> cnt = cnt + 1
>> 
>> fmt.Println("cnt:", cnt)
>> goroutineRunning.Add(1)
>> 
>> go func(p string) {
>>   goroutineRunning.Done()
>>   c.L.Lock() // critical section
>>   defer c.L.Unlock()
>> 
>>   fmt.Println("Registered and wait ... ")
>>   // HANG SPOT: can hang here 
>>   c.Wait()
>>   fmt.Println("afer goroutine wait for:", param)
>> 
>>   fn(p)
>> }(param)
>> fmt.Println("call wait before goroutine exit")
>> goroutineRunning.Wait()
>> fmt.Println("after wait, subscribe exits")
>> }
>> 
>> var clickRegistered sync.WaitGroup
>> 
>> for _, v := range []string{
>> "Maximizing window.",
>> "Displaying annoying dialog box!",
>> "Mouse clicked."} {
>> 
>> clickRegistered.Add(1)
>> 
>> subscribe(button.Clicked, v, func(s string) {
>> fmt.Println(s, "")
>> clickRegistered.Done()
>> })
>> }
>> 
>> fmt.Println("broadcast")
>> // is supposed to get the goroutines past HANG SPOT
>> button.Clicked.Broadcast()
>> time.Sleep(4000 * time.Millisecond)
>> 
>> clickRegistered.Wait()
>> }
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/d8148ef1-4d56-4b0b-a2d8-2992abe7a6c2n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/d8148ef1-4d56-4b0b-a2d8-2992abe7a6c2n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/BCE619ED-241A-4239-99E2-D8C20994EB0A%40ix.netcom.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/BCE619ED-241A-4239-99E2-D8C20994EB0A%40ix.netcom.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/68FB0F9A-E05F-4909-ABD6-13A05922DE86%40ix.netcom.com.

Re: [go-nuts] this code sometimes hangs

[robert engels]

Because the wait on the condition is in another Go routine, so the the 
Broadcast occurs before all of the routines are waiting. Broadcast only wakes 
up currently waiting routines.

> On Jul 18, 2024, at 9:01 AM, Lammie Jonson  wrote:
> 
> // I am not sure why this code sometimes deadlocks, but sometimes it doesn't.
> // Hopefully someone will have some suggestions 
> 
> package main
> 
> // ??? sometimes deadlocks
> 
> import (
> "fmt"
> "sync"
> "time"
> )
> 
> type Button struct {
> Clicked *sync.Cond
> }
> 
> func main() {
> button := Button{
> Clicked: sync.NewCond(&sync.Mutex{}),
> }
> 
> // running on goroutine every function that passed/registered
> // and wait, not exit until that goroutine is confirmed to be running
> subscribe := func(c *sync.Cond, param string, fn func(s string)) {
> var goroutineRunning sync.WaitGroup
> var cnt = 0
> cnt = cnt + 1
> 
> fmt.Println("cnt:", cnt)
> goroutineRunning.Add(1)
> 
> go func(p string) {
>   goroutineRunning.Done()
>   c.L.Lock() // critical section
>   defer c.L.Unlock()
> 
>   fmt.Println("Registered and wait ... ")
>   // HANG SPOT: can hang here 
>   c.Wait()
>   fmt.Println("afer goroutine wait for:", param)
> 
>   fn(p)
> }(param)
> fmt.Println("call wait before goroutine exit")
> goroutineRunning.Wait()
> fmt.Println("after wait, subscribe exits")
> }
> 
> var clickRegistered sync.WaitGroup
> 
> for _, v := range []string{
> "Maximizing window.",
> "Displaying annoying dialog box!",
> "Mouse clicked."} {
> 
> clickRegistered.Add(1)
> 
> subscribe(button.Clicked, v, func(s string) {
> fmt.Println(s, "")
> clickRegistered.Done()
> })
> }
> 
> fmt.Println("broadcast")
> // is supposed to get the goroutines past HANG SPOT
> button.Clicked.Broadcast()
> time.Sleep(4000 * time.Millisecond)
> 
> clickRegistered.Wait()
> }
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/d8148ef1-4d56-4b0b-a2d8-2992abe7a6c2n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/BCE619ED-241A-4239-99E2-D8C20994EB0A%40ix.netcom.com.

[go-nuts] Mutex profiling question

['Robert Engels' via golang-nuts]

I found this issue, https://github.com/golang/go/issues/57071 and tried running 
with 

GODEBUG=profileruntimelocks=1

but it made no difference in the output.

I am using Go 1.22.4 on OSX.

Any ideas how I can determine what is causing this high contention shown below:

(pprof) tree
Showing nodes accounting for 4440.35ms, 100% of 4440.35ms total
--+-
  flat  flat%   sum%cum   cum%   calls calls% + context 
 
--+-
 4324.17ms 97.38% 97.38%  4324.17ms 97.38%| 
runtime._LostContendedRuntimeLock
--+-
  116.17ms   100% |   sync.(*Map).Swap 
(inline)
  116.17ms  2.62%   100%   116.17ms  2.62%| sync.(*Mutex).Unlock
--+-
  116.17ms   100% |   
github.com/robaho/go-trader/internal/exchange.(*grpcServer).massquote
 0 0%   100%   116.17ms  2.62%| 
github.com/robaho/go-trader/internal/exchange.(*exchange).Quote
  116.17ms   100% |   
github.com/robaho/go-trader/internal/exchange.sendMarketData

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/25E80A1A-DFA8-4BA6-A49C-0F8F656DFA81%40me.com.

Re: [go-nuts] Importing non-module based code

['Robert Engels' via golang-nuts]

Thanks. Make sense.

> On Jul 16, 2024, at 9:00 AM, Jan Mercl <0xj...@gmail.com> wrote:
> 
> On Tue, Jul 16, 2024 at 3:57 PM Robert Engels  wrote:
> 
>> Yes, I just added that tag and pushed it, but @latest wouldn’t pick it up 
>> for some reason, needed to specify the release exactly using @v0.3.1
> 
> @latest is the "latest" known to the proxy server. It has some varying
> delay to update.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B17A12EB-A987-4AF1-9E81-EED04D6D11E1%40me.com.

Re: [go-nuts] Importing non-module based code

[robert engels]

I have go 1.21 specified in the go.mod, but I don’t think that mattered.

I added a tag to include my changes to my gocui and pushed (still did not add a 
go mod there), and it seems to work if I specify the tagged version - @latest 
did not pick it up, but maybe that uses the central registry which hadn’t 
updated yet. 

> On Jul 16, 2024, at 8:53 AM, Jason Phillips  
> wrote:
> 
> You don't mention what version of Go you're using but, note from the Go 1.22 
> release notes:
> 
> > go get is no longer supported outside of a module in the legacy GOPATH mode 
> > (that is, with GO111MODULE=off). Other build commands, such as go build and 
> > go test, will continue to work indefinitely for legacy GOPATH programs.
> 
> On Tuesday, July 16, 2024 at 9:51:02 AM UTC-4 Robert Engels wrote:
> Weird, even after I added the tag, using @latest did not pull the latest 
> code, I needed to specify the tag specifically @v0.3.1 
> 
> > On Jul 16, 2024, at 8:48 AM, Robert Engels  > > wrote: 
> > 
> > Thanks. Yea, because it needs a later tag now. I guess it never compiled 
> > after I changed go-trader to use modules. 
> > 
> >> On Jul 16, 2024, at 8:47 AM, Jan Mercl <0xj...@gmail.com 
> >> > wrote: 
> >> 
> >> On Tue, Jul 16, 2024 at 3:41 PM 'Robert Engels' via golang-nuts 
> >>  >> > wrote: 
> >> 
> >>> The go get works, and the code is there, but it isn’t seeing my code - it 
> >>> seems like it is using the original gocui which has since added module 
> >>> support. 
> >>> 
> >>> cmd/client/main.go:43:7: gui.Update undefined (type *gocui.Gui has no 
> >>> field or method Update) 
> >>  
> >>> 
> >>> but these methods are defined in the robaho fork of gocui 
> >> 
> >> Those methods are not defined: 
> >> https://pkg.go.dev/github.com/jroimartin/go...@v0.3.0 
> >> <https://pkg.go.dev/github.com/jroimartin/gocui@v0.3.0> 
> > 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/7c2824b8-4a89-4104-8223-c35ecdfa7074n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/7c2824b8-4a89-4104-8223-c35ecdfa7074n%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/14A28AF3-EFD6-4361-AF03-F36F006D2970%40ix.netcom.com.

Re: [go-nuts] Importing non-module based code

['Robert Engels' via golang-nuts]

Yes, I just added that tag and pushed it, but @latest wouldn’t pick it up for 
some reason, needed to specify the release exactly using @v0.3.1

> On Jul 16, 2024, at 8:55 AM, Jan Mercl <0xj...@gmail.com> wrote:
> 
> On Tue, Jul 16, 2024 at 3:50 PM Robert Engels  wrote:
>> 
>> Weird, even after I added the tag, using @latest did not pull the latest 
>> code, I needed to specify the tag specifically @v0.3.1
> 
> 
> I think I see the exported identifiers you need in v0.3.1, published
> 2018: https://pkg.go.dev/github.com/robaho/gocui@v0.3.1

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/E416A70C-BF68-408B-9F85-A8FF3F6FC6E7%40me.com.

Re: [go-nuts] Importing non-module based code

['Robert Engels' via golang-nuts]

Weird, even after I added the tag, using @latest did not pull the latest code, 
I needed to specify the tag specifically @v0.3.1

> On Jul 16, 2024, at 8:48 AM, Robert Engels  wrote:
> 
> Thanks. Yea, because it needs a later tag now. I guess it never compiled 
> after I changed go-trader to use modules.
> 
>> On Jul 16, 2024, at 8:47 AM, Jan Mercl <0xj...@gmail.com> wrote:
>> 
>> On Tue, Jul 16, 2024 at 3:41 PM 'Robert Engels' via golang-nuts
>>  wrote:
>> 
>>> The go get works, and the code is there, but it isn’t seeing my code - it 
>>> seems like it is using the original gocui which has since added module 
>>> support.
>>> 
>>> cmd/client/main.go:43:7: gui.Update undefined (type *gocui.Gui has no field 
>>> or method Update)
>> 
>>> 
>>> but these methods are defined in the robaho fork of gocui
>> 
>> Those methods are not defined:
>> https://pkg.go.dev/github.com/jroimartin/gocui@v0.3.0
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/A9055C11-FC9D-4445-8872-19492770ECC2%40me.com.

Re: [go-nuts] Importing non-module based code

['Robert Engels' via golang-nuts]

Thanks. Yea, because it needs a later tag now. I guess it never compiled after 
I changed go-trader to use modules.

> On Jul 16, 2024, at 8:47 AM, Jan Mercl <0xj...@gmail.com> wrote:
> 
> On Tue, Jul 16, 2024 at 3:41 PM 'Robert Engels' via golang-nuts
>  wrote:
> 
>> The go get works, and the code is there, but it isn’t seeing my code - it 
>> seems like it is using the original gocui which has since added module 
>> support.
>> 
>> cmd/client/main.go:43:7: gui.Update undefined (type *gocui.Gui has no field 
>> or method Update)
> 
>> 
>> but these methods are defined in the robaho fork of gocui
> 
> Those methods are not defined:
> https://pkg.go.dev/github.com/jroimartin/gocui@v0.3.0

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4E713B6F-857E-4208-AA3F-315D1405BAA1%40me.com.

Re: [go-nuts] Importing non-module based code

[robert engels]

I think I figured it out. I need to create a tag I guess? Since @latest doesn’t 
pull head ?

This seems to have changed, because I “think” this used to compile fine.



> On Jul 16, 2024, at 8:40 AM, 'Robert Engels' via golang-nuts 
>  wrote:
> 
> Has it changed so that Go repo that imports a from another repo that doesn’t 
> use modules no longer works? Maybe a recent change?
> 
> I have a repo github.com/robaho/go-trader 
> <http://github.com/robaho/go-trader> that imports githut.com/robaho/gocui 
> <http://githut.com/robaho/gocui> (which is a fork that doesn’t use modules).
> 
> I directly import from github.com/robaho/gocui 
> <http://github.com/robaho/gocui> and my go.mod has
> 
> github.com/robaho/gocui <http://github.com/robaho/gocui> v0.3.0
> 
> The go get works, and the code is there, but it isn’t seeing my code - it 
> seems like it is using the original gocui which has since added module 
> support.
> 
> cmd/client/main.go:43:7: gui.Update undefined (type *gocui.Gui has no field 
> or method Update)
> cmd/client/main.go:65:6: gui.Update undefined (type *gocui.Gui has no field 
> or method Update)
> cmd/client/main.go:78:6: gui.Update undefined (type *gocui.Gui has no field 
> or method Update)
> cmd/client/main.go:104:6: gui.Update undefined (type *gocui.Gui has no field 
> or method Update)
> cmd/client/main.go:166:6: gui.Update undefined (type *gocui.Gui has no field 
> or method Update)
> cmd/client/main.go:213:5: v.MaxLines undefined (type *gocui.View has no field 
> or method MaxLines)
> cmd/client/main.go:226:5: v.MaxLines undefined (type *gocui.View has no field 
> or method MaxLines)
> cmd/client/main.go:234:5: v.MaxLines undefined (type *gocui.View has no field 
> or method MaxLines)
> cmd/client/main.go:251:5: v.Editor undefined (type *gocui.View has no field 
> or method Editor)
> cmd/client/main.go:256:5: g.Update undefined (type *gocui.Gui has no field or 
> method Update)
> cmd/client/main.go:256:5: too many errors
> 
> but these methods are defined in the robaho fork of gocui
> 
> I can add module support to my fork but it seems like this shouldn't be 
> necessary… ???
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/5F9D9A4C-5437-46BA-9ED2-681FA67EDC1D%40me.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/5F9D9A4C-5437-46BA-9ED2-681FA67EDC1D%40me.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5FB8083E-06B7-481D-8845-4A1410708654%40ix.netcom.com.

[go-nuts] Importing non-module based code

['Robert Engels' via golang-nuts]

Has it changed so that Go repo that imports a from another repo that doesn’t 
use modules no longer works? Maybe a recent change?

I have a repo github.com/robaho/go-trader  
that imports githut.com/robaho/gocui  (which is 
a fork that doesn’t use modules).

I directly import from github.com/robaho/gocui  
and my go.mod has

github.com/robaho/gocui v0.3.0

The go get works, and the code is there, but it isn’t seeing my code - it seems 
like it is using the original gocui which has since added module support.

cmd/client/main.go:43:7: gui.Update undefined (type *gocui.Gui has no field or 
method Update)
cmd/client/main.go:65:6: gui.Update undefined (type *gocui.Gui has no field or 
method Update)
cmd/client/main.go:78:6: gui.Update undefined (type *gocui.Gui has no field or 
method Update)
cmd/client/main.go:104:6: gui.Update undefined (type *gocui.Gui has no field or 
method Update)
cmd/client/main.go:166:6: gui.Update undefined (type *gocui.Gui has no field or 
method Update)
cmd/client/main.go:213:5: v.MaxLines undefined (type *gocui.View has no field 
or method MaxLines)
cmd/client/main.go:226:5: v.MaxLines undefined (type *gocui.View has no field 
or method MaxLines)
cmd/client/main.go:234:5: v.MaxLines undefined (type *gocui.View has no field 
or method MaxLines)
cmd/client/main.go:251:5: v.Editor undefined (type *gocui.View has no field or 
method Editor)
cmd/client/main.go:256:5: g.Update undefined (type *gocui.Gui has no field or 
method Update)
cmd/client/main.go:256:5: too many errors

but these methods are defined in the robaho fork of gocui

I can add module support to my fork but it seems like this shouldn't be 
necessary… ???

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5F9D9A4C-5437-46BA-9ED2-681FA67EDC1D%40me.com.

Re: [go-nuts] Benchmark "overall" operation

['Robert Engels' via golang-nuts]

> The usual way is to factor out any parts that can skew the results
> before the benchmark loop and call b.ResetTimer() just before entering
> the 'for i := 0; i < b.N; i++ {' loop.

That isn’t applicable here. It is the performing of the operation that gets 
progressively slower. So when comparing across implementations, you want to use 
a fixed number of operations, and time the overall execution time, creating a 
time/op output.

> I don't think it's useful. The testing package does IMO the right

Fair enough, but if you were writing a benchmark against sorting algorithms, 
how would you benchmark say a bubble sort against a quicksort. I guess you 
would insert all of the items and then ResetTimer(), and then still you would 
get the total sort time, not the sort time per item.

So then you need to create:

Sort1000 items
Sort1 items
Sort10 items

but the output it still the time it takes to run the sort, not the sort time 
amortized per item.

Making it much more difficult to compare different algorithms and different 
container sizes.

> On Jul 8, 2024, at 3:51 PM, Jan Mercl <0xj...@gmail.com> wrote:
> 
> On Mon, Jul 8, 2024 at 10:36 PM Robert Engels  wrote:
> 
>> Is there anyway to do what I want?
> 
> The usual way is to factor out any parts that can skew the results
> before the benchmark loop and call b.ResetTimer() just before entering
> the 'for i := 0; i < b.N; i++ {' loop.
> 
>> If not, maybe a useful addition, see 
>> https://javadoc.io/doc/org.openjdk.jmh/jmh-core/latest/org/openjdk/jmh/annotations/OperationsPerInvocation.html
>>  for a similar Java api.
> 
> I don't think it's useful. The testing package does IMO the right
> thing. The task of the benchmark loop is to always do the same
> unit/amount of work b.N times. Only then it all together makes sense
> and can produce meaningful data about the unit/amount. It follows that
> measuring O(b.N^2) algorithms cannot work very well. But you can often
> measure b.N times a O(K^2) work using a constant K.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/FABCC32C-1918-4E60-87A2-AB4F8E0E7CAD%40me.com.

Re: [go-nuts] Benchmark "overall" operation

['Robert Engels' via golang-nuts]

Understood.

Is there anyway to do what I want? If not, maybe a useful addition, see 
https://javadoc.io/doc/org.openjdk.jmh/jmh-core/latest/org/openjdk/jmh/annotations/OperationsPerInvocation.html
 
<https://javadoc.io/doc/org.openjdk.jmh/jmh-core/latest/org/openjdk/jmh/annotations/OperationsPerInvocation.html>
 for a similar Java api.

The reason this is helpful, is that some algorithms will degrade on subsequent 
(e.g. the first order insertion is really fast, but they get slower and slower 
because the dev used an O(n^2) design - a test that only goes to b.N may not 
hit it.

It is also useful when comparing performance metrics for operations like these 
across languages (similar testing framework in a non-Go language).

So you need to be able to test the add operation but calculate the average time 
per op across the test with a minimum number of elements.

Is there any canonical way of doing this? 

> On Jul 8, 2024, at 3:22 PM, Jan Mercl <0xj...@gmail.com> wrote:
> 
> On Mon, Jul 8, 2024 at 10:08 PM 'Robert Engels' via golang-nuts
>  wrote:
> 
>> Given this code (which I know is not “correct”):
> 
> Setting b.N in the benchmark code has no specified behavior. The
> actual behavior at the moment ignores such changes.
> 
> The algorithm used by the testing package adjusts b.N in an attempt to
> stabilize the results under certain conditions. Ignoring this guidance
> makes all attempts of that algorithm futile. And as it assumes the
> benchmark body was executed b.N times anyway, it computes garbage.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAA40n-UK65F%3DNp%2BkS59w_y-G%2BSQeFoh5V9b8Ezgt9TkGuzx-7g%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/E13E028C-9962-4D74-9D19-3615303CB6ED%40me.com.

Re: [go-nuts] Benchmark "overall" operation

[robert engels]

Actually, if I add

b.Log("time per op",float64(b.Elapsed().Microseconds())/float64(N_ORDERS*2))

I get a different value than what benchmark reports, so I am certain setting 
b.N is not correct.

> On Jul 8, 2024, at 3:08 PM, 'Robert Engels' via golang-nuts 
>  wrote:
> 
> Given this code (which I know is not “correct”):
> 
> func BenchmarkOrders(b *testing.B) {
>var ob = orderBook{}
>var inst = Equity{}
>var ex = testExchangeClient{}
> 
>const N_ORDERS = 100
> 
>b.ResetTimer()
> 
>b.N = N_ORDERS
> 
>for i:=0;ivar o1 = LimitOrder(inst, Buy, NewDecimal("100"), NewDecimal("10"))
>o1.ExchangeId = fmt.Sprint(i)
>var s1 = sessionOrder{ex, o1, time.Now()}
>ob.add(s1)
>}
>for i:=0;ivar o1 = LimitOrder(inst, Sell, NewDecimal("100"), NewDecimal("10"))
>o1.ExchangeId = "S"+fmt.Sprint(i)
>var s1 = sessionOrder{ex, o1, time.Now()}
>ob.add(s1)
>}
> 
> }
> 
> Instead of setting b.N, is there a better way to get a “benchmark” for the 
> logical operation.
> 
> In this case, the number of operations is 2 * N_ORDERS, so I would like to 
> time the entire process and output in the benchmark format the time per “op”.
> 
> It seems to work setting b.N, but I’m sure this isn’t ideal.
> 
> In Java JMH, you have the ability to set the “number of operations” performed 
> during the test - I’m looking for something similar.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/E5D80591-F819-4C4D-9E18-6E2438B269FB%40me.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/63E0AA90-54AB-480F-828A-44C7376CB351%40ix.netcom.com.

[go-nuts] Benchmark "overall" operation

['Robert Engels' via golang-nuts]

Given this code (which I know is not “correct”):

func BenchmarkOrders(b *testing.B) {
var ob = orderBook{}
var inst = Equity{}
var ex = testExchangeClient{}

const N_ORDERS = 100

b.ResetTimer()

b.N = N_ORDERS

for i:=0;ihttps://groups.google.com/d/msgid/golang-nuts/E5D80591-F819-4C4D-9E18-6E2438B269FB%40me.com.

Re: [go-nuts] Does data race with only multiple goroutine parallel i=1 have an impact on the execution results?

[Robert Engels]

I corrected myself. On Jul 6, 2024, at 10:34 AM, peterGo  wrote:On Saturday, July 6, 2024 at 11:21:46 AM UTC-4 Robert Engels wrote:That is not a data race. The wait group is a synchronization barrier. ==WARNING: DATA RACEWrite at 0x00c14158 by goroutine 8:  main.main.func1()      /home/peter/racer.go:12 +0x84Previous write at 0x00c14158 by goroutine 13:  main.main.func1()      /home/peter/racer.go:12 +0x84Goroutine 8 (running) created at:  main.main()      /home/peter/racer.go:10 +0x78Goroutine 13 (running) created at:  main.main()      /home/peter/racer.go:10 +0x78==1Found 1 data race(s)peter 



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e814204e-5461-4d65-a7e7-99feaf94294an%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/D8F05B47-12B0-48D3-99D2-2A920B133BD9%40ix.netcom.com.

Re: [go-nuts] Does data race with only multiple goroutine parallel i=1 have an impact on the execution results?

[Robert Engels]

Sorry and the outer loop is the only reader. It probably reports it as a race though since there are multiple writers - but they all write the same value. Technically still a data race depending on if the word splits when writing. On Jul 6, 2024, at 10:20 AM, Robert Engels  wrote:That is not a data race. The wait group is a synchronization barrier. On Jul 6, 2024, at 9:34 AM, 'qiu laidongfeng2' via golang-nuts  wrote:I know not to write code with data races?The problem is that I find that this code always outputs 1 on the amd64 machine,I am not sure what effect this data competition has on the execution result,whether it is an inevitable result of the amd64 machine to always output 1 or I am lucky to encounter that it is always output 1, so I ask.在2024年7月6日星期六 UTC+8 22:20:57 写道：On Sat, Jul 6, 2024 at 7:04 AM 'qiu laidongfeng2' via golang-nuts
<golan...@googlegroups.com> wrote:
>
> Does data race in this program affect execution results?
> In amd64, it always output1.

If you want to write code like this, use the sync/atomic package.
Don't write code with data races.

Ian




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6a2c3ebf-b5e9-4a18-bcf3-d7a4b5fc24a4n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/40CF9EFD-6B8B-431A-ABD8-DE994F044156%40ix.netcom.com.

Re: [go-nuts] Does data race with only multiple goroutine parallel i=1 have an impact on the execution results?

[Robert Engels]

That is not a data race. The wait group is a synchronization barrier. On Jul 6, 2024, at 9:34 AM, 'qiu laidongfeng2' via golang-nuts  wrote:I know not to write code with data races?The problem is that I find that this code always outputs 1 on the amd64 machine,I am not sure what effect this data competition has on the execution result,whether it is an inevitable result of the amd64 machine to always output 1 or I am lucky to encounter that it is always output 1, so I ask.在2024年7月6日星期六 UTC+8 22:20:57 写道：On Sat, Jul 6, 2024 at 7:04 AM 'qiu laidongfeng2' via golang-nuts
 wrote:
>
> Does data race in this program affect execution results?
> In amd64, it always output1.

If you want to write code like this, use the sync/atomic package.
Don't write code with data races.

Ian




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6a2c3ebf-b5e9-4a18-bcf3-d7a4b5fc24a4n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/A8A9A454-827C-4F17-BC10-2B5BA638A799%40ix.netcom.com.

Re: [go-nuts] More excellent work by the Go team!

[robert engels]

Thanks for point it out Brian. I must of had some background interference. I 
re-ran the tests, Go was the same, Java is

Benchmark   Mode  Cnt ScoreError  Units
TestJavaDispatch.TestArrayDispatch  avgt9  4367.196 ± 50.227  ns/op
TestJavaDispatch.TestDispatch   avgt9 0.942 ±  0.005  ns/op
TestJavaDispatch.TestIDispatch  avgt9 1.341 ±  0.040  ns/op

So the error is much less.

> On Jun 21, 2024, at 2:14 PM, Robert Engels  wrote:
> 
> Yep. Haven’t dig into it. I assume it has to do with allocations. That it is 
> 2x faster than Go is somewhat suspect as well. 
> 
> I was more sharing for the change in Go performance. 
> 
>> On Jun 21, 2024, at 2:11 PM, 'Brian Candler' via golang-nuts 
>>  wrote:
>> 
>> > 4775.994 ± 13310.309
>> 
>> Those are pretty wide error bars though...
>> 
>> On Friday 21 June 2024 at 17:15:05 UTC+1 Robert Engels wrote:
>> I had the opportunity to revisit my interface dispatch benchmark 
>> (https://github.com/robaho/go-dispatch-test 
>> <https://github.com/robaho/go-dispatch-test>).
>> 
>> Previously, the dispatching using pointer receivers was nearly 3x slower 
>> (i.e. 5.5 ns to 12.6 ns):
>> 
>> goos: darwin
>> goarch: amd64
>> pkg: github.com/robaho/go-dispatch-test 
>> <http://github.com/robaho/go-dispatch-test>
>> BenchmarkIDispatch-835.56 ns/op
>> BenchmarkDispatch-8 20   1.57 ns/op
>> BenchmarkArrayParms-8 20  9265 ns/op
>> PASS
>> 
>> and
>> 
>> goos: darwin
>> goarch: amd64
>> pkg: github.com/robaho/go-dispatch-test 
>> <http://github.com/robaho/go-dispatch-test>
>> BenchmarkIDispatch-81   12.6 ns/op
>> BenchmarkDispatch-8 20   1.57 ns/op
>> PASS
>> 
>> 
>> With the latest Go 1.22.4, the performance difference has reversed ! (tests 
>> have been combined for simplicity).
>> 
>> goos: darwin
>> goarch: amd64
>> pkg: github.com/robaho/go-dispatch-test 
>> <http://github.com/robaho/go-dispatch-test>
>> cpu: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>> BenchmarkIDispatch-85352637462.118 ns/op
>> BenchmarkIDispatchPtr-8 6757212441.773 ns/op
>> BenchmarkDispatch-8 10   0.6953 ns/op
>> BenchmarkArrayParms-8 184212  6440 ns/op
>> PASS
>> 
>> Pretty impressive work. The interface based dispatch is almost equal to the 
>> Java version now:
>> 
>> Benchmark   Mode  Cnt Score   Error  Units
>> TestJavaDispatch.TestArrayDispatch  avgt3  4775.994 ± 13310.309  ns/op
>> TestJavaDispatch.TestDispatch   avgt3 0.941 ± 0.088  ns/op
>> TestJavaDispatch.TestIDispatch  avgt3 1.331 ± 0.146  ns/op
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/57b5e091-f309-4d5f-b102-f58c3d344776n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/57b5e091-f309-4d5f-b102-f58c3d344776n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/6E7B8F20-45AE-462F-85FD-7A30A77806C7%40ix.netcom.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/6E7B8F20-45AE-462F-85FD-7A30A77806C7%40ix.netcom.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/D940D5A5-7313-4E2F-B048-E32AA6B402BE%40ix.netcom.com.

Re: [go-nuts] Re: More excellent work by the Go team!

[Robert Engels]

Yep. Haven’t dig into it. I assume it has to do with allocations. That it is 2x faster than Go is somewhat suspect as well. I was more sharing for the change in Go performance. On Jun 21, 2024, at 2:11 PM, 'Brian Candler' via golang-nuts  wrote:> 4775.994 ± 13310.309Those are pretty wide error bars though...On Friday 21 June 2024 at 17:15:05 UTC+1 Robert Engels wrote:I had the opportunity to revisit my interface dispatch benchmark (https://github.com/robaho/go-dispatch-test).Previously, the dispatching using pointer receivers was nearly 3x slower (i.e. 5.5 ns to 12.6 ns):goos: darwingoarch: amd64pkg: github.com/robaho/go-dispatch-testBenchmarkIDispatch-835.56 ns/opBenchmarkDispatch-8 20   1.57 ns/opBenchmarkArrayParms-8 20  9265 ns/opPASSandgoos: darwingoarch: amd64pkg: github.com/robaho/go-dispatch-testBenchmarkIDispatch-81   12.6 ns/opBenchmarkDispatch-8 20   1.57 ns/opPASSWith the latest Go 1.22.4, the performance difference has reversed ! (tests have been combined for simplicity).goos: darwingoarch: amd64pkg: github.com/robaho/go-dispatch-testcpu: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHzBenchmarkIDispatch-85352637462.118 ns/opBenchmarkIDispatchPtr-8 6757212441.773 ns/opBenchmarkDispatch-8 10   0.6953 ns/opBenchmarkArrayParms-8 184212  6440 ns/opPASSPretty impressive work. The interface based dispatch is almost equal to the Java version now:Benchmark   Mode  Cnt Score   Error  UnitsTestJavaDispatch.TestArrayDispatch  avgt3  4775.994 ± 13310.309  ns/opTestJavaDispatch.TestDispatch   avgt3 0.941 ± 0.088  ns/opTestJavaDispatch.TestIDispatch  avgt3 1.331 ± 0.146  ns/op



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/57b5e091-f309-4d5f-b102-f58c3d344776n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6E7B8F20-45AE-462F-85FD-7A30A77806C7%40ix.netcom.com.

[go-nuts] More excellent work by the Go team!

['Robert Engels' via golang-nuts]

I had the opportunity to revisit my interface dispatch benchmark 
(https://github.com/robaho/go-dispatch-test 
).

Previously, the dispatching using pointer receivers was nearly 3x slower (i.e. 
5.5 ns to 12.6 ns):

goos: darwin
goarch: amd64
pkg: github.com/robaho/go-dispatch-test
BenchmarkIDispatch-835.56 ns/op
BenchmarkDispatch-8 20   1.57 ns/op
BenchmarkArrayParms-8 20  9265 ns/op
PASS

and

goos: darwin
goarch: amd64
pkg: github.com/robaho/go-dispatch-test
BenchmarkIDispatch-81   12.6 ns/op
BenchmarkDispatch-8 20   1.57 ns/op
PASS


With the latest Go 1.22.4, the performance difference has reversed ! (tests 
have been combined for simplicity).

goos: darwin
goarch: amd64
pkg: github.com/robaho/go-dispatch-test
cpu: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
BenchmarkIDispatch-85352637462.118 ns/op
BenchmarkIDispatchPtr-8 6757212441.773 ns/op
BenchmarkDispatch-8 10   0.6953 ns/op
BenchmarkArrayParms-8 184212  6440 ns/op
PASS

Pretty impressive work. The interface based dispatch is almost equal to the 
Java version now:

Benchmark   Mode  Cnt Score   Error  Units
TestJavaDispatch.TestArrayDispatch  avgt3  4775.994 ± 13310.309  ns/op
TestJavaDispatch.TestDispatch   avgt3 0.941 ± 0.088  ns/op
TestJavaDispatch.TestIDispatch  avgt3 1.331 ± 0.146  ns/op

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/E96FF86B-1EF9-4678-8DE4-11D4C7080C0E%40me.com.

Re: [go-nuts] Comparison of pointers to distinct zero-sized variables

[Robert Engels]

What I asked did not require the first, I was referring more about the latter. So regardless of how it is implemented it is always false. On Jun 20, 2024, at 7:39 AM, Axel Wagner  wrote:On Thu, 20 Jun 2024 at 14:21, Robert Engels <reng...@ix.netcom.com> wrote:I would add that I think the Go team should strive to remove as much “implementation defined” or “undefined” from the spec as possible. Forgo the optimizations. It’s this language that makes C++ such a pain to work with especially in a cross platform manner. I think you can safely narrow the spec in this area without issue - since it would have been very difficult to use anyway. Maybe some highly specialized code that relied on a particular Go implementation might. I strongly disagree. There is a lot of software out there that relies on zero-sized fields to achieve certain semantic behaviours (like preventing copying or using unkeyed struct literals) or alignments. They all assume that the size of a struct is not impacted by that. Without this "optimization" that would no longer be true and every zero-sized field must take at least one byte of memory, to be able to guarantee that they have different addresses. That is just not going to happen.On the other hand, removing the restriction in the other direction and just always making it false is what I suggested last time and there are reasons in that issue, why that isn't going to happen either. On Jun 20, 2024, at 7:12 AM, 'Axel Wagner' via golang-nuts <golang-nuts@googlegroups.com> wrote:On Thu, 20 Jun 2024 at 13:48, Oliver Eikemeier <eikeme...@fillmore-labs.com> wrote:Am 20.06.2024 um 13:24 schrieb Axel Wagner <axel.wagner...@googlemail.com>:We can see that 1. `unsafe.Pointer` is definitionally a pointer type, 2. in your example, the two `unsafe.Pointers` do not point to the same variable and do not have the value `nil` and 3. are not pointing at distinct zero-sized variables. So their comparison should be `false`, which is what your example observes.Does it (Go Playground)?func f5() {	var (		a  struct{}		b  int		aa = (*int)(unsafe.Pointer(&a))		ba = (*int)(unsafe.Pointer(&b))		eq = aa == ba	)	println("&a:", aa, reflect.TypeOf(aa).String())	println("&b:", ba, reflect.TypeOf(ba).String())	println("&a == &b:", eq)	cmpIntPtr(aa, ba)}func cmpIntPtr(p1 *int, p2 *int) {	fmt.Println("p1 == p2:", p1 == p2)}&a: 0xc466f8 *int
&b: 0xc466f8 *int
&a == &b: false
p1 == p2: trueInterestingly, the spec does not forbid that (or specifies it as implementation-defined), though the (implementation-defined) unsafe.Pointer rules
 do, as using `unsafe.Pointer` to convert between pointers of different 
base types is only allowed if the target variable is at least as large 
as the source variable and both share an equivalent memory layout.I
 think the fact that the spec does not put any limits on the allowed 
behaviour of converting between `unsafe.Pointer` and other pointer types
 is a clear oversight that should be corrected (perhaps it should really
 be "The effect of converting between Pointer and uintptr is 
implementation-defined" should really say "and other types"). But I 
don't think it's a surprising fact that you can make a program behave in almost arbitrary ways by doing that. I’m advocating for at least a FAQ article,I tend to agree, though I'm not sure how to phrase that, beyond saying "do not make any assumptions about the identity pointers to zero-sized variables or with zero-sized base types or that where derived from one of those" and I'm not sure how helpful that is.It seems like a frequently asked question to me ;)It is. But note that every time it is asked, it leads to considerable discussion, which seems to strongly suggest that it is hard to give a simple, unambiguous answer that people find satisfying.I think you should acknowledge that the spec *is* already trying to be very clear about this. The fact that you can poke holes in it is because writing a spec unambiguously is hard. But the two quoted phrases are there in the best attempt to clearly state that you can not rely on any assumptions about pointers to zero-sized variables - within the somewhat abstract framework of the spec. Feel free to propose a better one, I'm not actually arguing against that.And I'm also not arguing against an FAQ entry. Ideally, someone would suggest a phrasing that is sufficiently broad and unambiguous and clear to cover any version of this question people might ask, while also not contradicting the spec. but also think the specification should be adapted, for clarity but also for the fact that only one pointer pointing to a zero-sized variable can compare differently to anything over time, even things having the same address value.Note that "address value" is not something that exists within the spec. As for 

Re: [go-nuts] Comparison of pointers to distinct zero-sized variables

[Robert Engels]

I would add that I think the Go team should strive to remove as much “implementation defined” or “undefined” from the spec as possible. Forgo the optimizations. It’s this language that makes C++ such a pain to work with especially in a cross platform manner. I think you can safely narrow the spec in this area without issue - since it would have been very difficult to use anyway. Maybe some highly specialized code that relied on a particular Go implementation might. On Jun 20, 2024, at 7:12 AM, 'Axel Wagner' via golang-nuts  wrote:On Thu, 20 Jun 2024 at 13:48, Oliver Eikemeier  wrote:Am 20.06.2024 um 13:24 schrieb Axel Wagner :We can see that 1. `unsafe.Pointer` is definitionally a pointer type, 2. in your example, the two `unsafe.Pointers` do not point to the same variable and do not have the value `nil` and 3. are not pointing at distinct zero-sized variables. So their comparison should be `false`, which is what your example observes.Does it (Go Playground)?func f5() {	var (		a  struct{}		b  int		aa = (*int)(unsafe.Pointer(&a))		ba = (*int)(unsafe.Pointer(&b))		eq = aa == ba	)	println("&a:", aa, reflect.TypeOf(aa).String())	println("&b:", ba, reflect.TypeOf(ba).String())	println("&a == &b:", eq)	cmpIntPtr(aa, ba)}func cmpIntPtr(p1 *int, p2 *int) {	fmt.Println("p1 == p2:", p1 == p2)}&a: 0xc466f8 *int
&b: 0xc466f8 *int
&a == &b: false
p1 == p2: trueInterestingly, the spec does not forbid that (or specifies it as implementation-defined), though the (implementation-defined) unsafe.Pointer rules
 do, as using `unsafe.Pointer` to convert between pointers of different 
base types is only allowed if the target variable is at least as large 
as the source variable and both share an equivalent memory layout.I
 think the fact that the spec does not put any limits on the allowed 
behaviour of converting between `unsafe.Pointer` and other pointer types
 is a clear oversight that should be corrected (perhaps it should really
 be "The effect of converting between Pointer and uintptr is 
implementation-defined" should really say "and other types"). But I 
don't think it's a surprising fact that you can make a program behave in almost arbitrary ways by doing that. I’m advocating for at least a FAQ article,I tend to agree, though I'm not sure how to phrase that, beyond saying "do not make any assumptions about the identity pointers to zero-sized variables or with zero-sized base types or that where derived from one of those" and I'm not sure how helpful that is.It seems like a frequently asked question to me ;)It is. But note that every time it is asked, it leads to considerable discussion, which seems to strongly suggest that it is hard to give a simple, unambiguous answer that people find satisfying.I think you should acknowledge that the spec *is* already trying to be very clear about this. The fact that you can poke holes in it is because writing a spec unambiguously is hard. But the two quoted phrases are there in the best attempt to clearly state that you can not rely on any assumptions about pointers to zero-sized variables - within the somewhat abstract framework of the spec. Feel free to propose a better one, I'm not actually arguing against that.And I'm also not arguing against an FAQ entry. Ideally, someone would suggest a phrasing that is sufficiently broad and unambiguous and clear to cover any version of this question people might ask, while also not contradicting the spec. but also think the specification should be adapted, for clarity but also for the fact that only one pointer pointing to a zero-sized variable can compare differently to anything over time, even things having the same address value.Note that "address value" is not something that exists within the spec. As for clarifying the spec here, maybe. I do think the behavior is covered, as I outlined above. And as above, I'm not sure how to clarify it further, while still leaving up the space we want to left open.I vote for making this openness explicit, meaning that even the optimizer is free to make assumptions that do not hold during runtime. So if you have a pointer derived from something pointing to a zero-size type it can compare with different results over time, even false when having the same value. I do not believe that is what’s currently in the spec. CheersOliver



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGOWoQYrQ9qm_9%2BdyvEObJV3JP%2BhwwFj9%2Bgo%3DqusRVL7A%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.

Re: [go-nuts] Re: Breaking an io.Pipe loop

[Robert Engels]

Nothing is writing and closing StdInR so it will never complete. On Jun 17, 2024, at 9:54 PM, Salvatore Domenick Desiano  wrote:A bunch of typos. The last paragraph should have been:This program deadlocks after 5 seconds. The three deadlocked Go routines are the explicit go func() (stuck on io.Copy() because cmdStdoutR never closes), exec.Command's internal Go routine copying data to stdin also (stuck on io.Copy() because cmdStdinR never closes), and the main routine (stuck on cmd.Wait()).



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEQs7S9e8bmCd7cFAzDTu5g3rUQnuT7iaKo%2BLRR-J1nH%3D9%2BNBw%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5F4E3FB8-03A4-4DC6-B566-4330A2F2DC8E%40ix.netcom.com.

Re: [go-nuts] Breaking an io.Pipe loop

[Robert Engels]

Code?On Jun 17, 2024, at 9:46 PM, Salvatore Domenick Desiano  wrote:I've got some code that is basically Expect-lite written in Go. I've got a deadlock and I'm hoping someone here can help me untangle it. I posted this yesterday but at the time I thought the Expect part of the code wasn't relevant and I left it out. I now believe that part of the code is the problem.I have:exec.Command() with stdin and stdout replaced with io.Pipe()a Go routine listening to stdout, watching for strings, and writing to stdinI know that Wait() won't return until stdin is closed. The problem is that I can't tell when I need to close stdin. Since Wait() doesn't return, the main thread can't do it. For some reason stdout doesn't get closed even with the program exits (stdout.Read() is part of the deadlock) so the Expect Go routine can't do it.The only thing I can think of is another Go routine monitoring cmd.ProcessState but that doesn't feel right.What am I missing?Thank you!-- Salvatoresmile.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEQs7S8M_KTESzGPfaQsEHi9gVc1O_M5DAS4nKJ0SDn_Mqm3Tg%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/B321E9C9-AEE8-4CC1-8AC5-02D34B2A8E12%40ix.netcom.com.

Re: [go-nuts] Fwd: cmd.Wait & io.Pipe & SIGKILL

[Robert Engels]

It hangs because cmd.Run() waits for the process to complete - and with nothing reading the pipe it will never complete. On Jun 16, 2024, at 9:33 PM, Robert Engels  wrote:It looks like you don’t have anything reading from the pipe so it’s going to hang. On Jun 16, 2024, at 8:54 PM, Salvatore Domenick Desiano  wrote:Simpler self-contained example, same result:func main() {	cmdName := "/bin/bash"	cmdArgs := []string{"-c", "while true; do echo step; sleep 1; done"}cmdStdoutR, cmdStdoutW := io.Pipe()	cmd := exec.Command(cmdName, cmdArgs...)	cmd.Stdout = cmdStdoutW	if err := cmd.Run(); err != nil {		fmt.Println(err)	}	fmt.Println("DONE")cmdStdoutR = cmdStdoutR}For the sake of completeness I'll mention that this runs fine if cmd.Stdout is not assigned, which points to exec.awaitGoroutines() as being the culprit.FWIW, I have no direct evidence that the pipes aren't being closed on the SIGKILL. That said, the (exec-internal) goroutines are hung on io.Copy(). I expect that if the process's output pipe was actually closed the io.Copy() would return, the goroutine would complete, and the deadlock would not happen.-- Salvatoresmile.On Sun, Jun 16, 2024 at 7:58 PM Salvatore Domenick Desiano <neard...@gmail.com> wrote:I hope this isn't a red herring but in constructing a self-contained example I triggered a deadlock. Perhaps the deadlock is the answer to my question or perhaps it is another issue. Either way this code does not seem malformed:func main() {	cmdName := "/bin/bash"	cmdArgs := []string{		"-c",		"while true; do echo step; sleep 1; done",	}	cmdStdinR, cmdStdinW := io.Pipe()	cmdStdoutR, cmdStdoutW := io.Pipe()cmdStderrR, cmdStderrW := io.Pipe()ctx, cancelCmd := context.WithCancel(context.Background())	cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)	cmd.Stdin = cmdStdinR	cmd.Stdout = cmdStdoutW	cmd.Stderr = cmdStderrW	if err := cmd.Start(); err != nil {		fmt.Println(err)	}cmdStdinW.Close()if err := cmd.Wait(); err != nil {fmt.Println(err)}fmt.Println("DONE")	cancelCmd = cancelCmdcmdStdoutR = cmdStdoutRcmdStderrR = cmdStderrR}If you run this code and SIGKILL the bash process, go flags it as a deadlock and panics. FWIW, this also happens if there are goroutines monitoring cmdStdoutR and cmdStderrR.What am I missing?-- Salvatoresmile.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEQs7S-Gwuvh8MJ48Nv1Cwd7miHiUX4RdwKjqmeLqMBcWYYgJQ%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/BB6ADA54-22F5-44AC-AB9B-149308C5FA56%40ix.netcom.com.

Re: [go-nuts] Fwd: cmd.Wait & io.Pipe & SIGKILL

[Robert Engels]

It looks like you don’t have anything reading from the pipe so it’s going to hang. On Jun 16, 2024, at 8:54 PM, Salvatore Domenick Desiano  wrote:Simpler self-contained example, same result:func main() {	cmdName := "/bin/bash"	cmdArgs := []string{"-c", "while true; do echo step; sleep 1; done"}cmdStdoutR, cmdStdoutW := io.Pipe()	cmd := exec.Command(cmdName, cmdArgs...)	cmd.Stdout = cmdStdoutW	if err := cmd.Run(); err != nil {		fmt.Println(err)	}	fmt.Println("DONE")cmdStdoutR = cmdStdoutR}For the sake of completeness I'll mention that this runs fine if cmd.Stdout is not assigned, which points to exec.awaitGoroutines() as being the culprit.FWIW, I have no direct evidence that the pipes aren't being closed on the SIGKILL. That said, the (exec-internal) goroutines are hung on io.Copy(). I expect that if the process's output pipe was actually closed the io.Copy() would return, the goroutine would complete, and the deadlock would not happen.-- Salvatoresmile.On Sun, Jun 16, 2024 at 7:58 PM Salvatore Domenick Desiano  wrote:I hope this isn't a red herring but in constructing a self-contained example I triggered a deadlock. Perhaps the deadlock is the answer to my question or perhaps it is another issue. Either way this code does not seem malformed:func main() {	cmdName := "/bin/bash"	cmdArgs := []string{		"-c",		"while true; do echo step; sleep 1; done",	}	cmdStdinR, cmdStdinW := io.Pipe()	cmdStdoutR, cmdStdoutW := io.Pipe()cmdStderrR, cmdStderrW := io.Pipe()ctx, cancelCmd := context.WithCancel(context.Background())	cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)	cmd.Stdin = cmdStdinR	cmd.Stdout = cmdStdoutW	cmd.Stderr = cmdStderrW	if err := cmd.Start(); err != nil {		fmt.Println(err)	}cmdStdinW.Close()if err := cmd.Wait(); err != nil {fmt.Println(err)}fmt.Println("DONE")	cancelCmd = cancelCmdcmdStdoutR = cmdStdoutRcmdStderrR = cmdStderrR}If you run this code and SIGKILL the bash process, go flags it as a deadlock and panics. FWIW, this also happens if there are goroutines monitoring cmdStdoutR and cmdStderrR.What am I missing?-- Salvatoresmile.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEQs7S-Gwuvh8MJ48Nv1Cwd7miHiUX4RdwKjqmeLqMBcWYYgJQ%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5A7A385F-BD84-4973-BEB9-63F7231BEAB7%40ix.netcom.com.

Re: [go-nuts] Re: coroutine size

[Robert Engels]

I don’t think that is possible. You can have stackless coroutines, but if a coroutine has no state at all it would simply be implements as a method call chain without parameters in an executor. I doubt these are very useful. On Jun 15, 2024, at 10:08 PM, cheng dong  wrote:ahh, i really want to say is, can golang offer user a stateless coroutine. On Saturday, June 15, 2024 at 11:16:02 PM UTC+8 cheng dong wrote:sorry, i found that 500k stack size might be nesessary for a coroutine, so its not a problem.On Saturday, June 15, 2024 at 10:24:18 PM UTC+8 cheng dong wrote:maybe coroutine could be implented inside a g in future hopefullyOn Saturday, June 15, 2024 at 10:23:01 PM UTC+8 cheng dong wrote:thanks to runtime.newcoro/coroswitch and now we could do more things with coroutine. but it seems every newcoro create a full g which is 448B in size. maybe it is too heavy for a coroutine both in size and in schedule granularity ?



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1d0dd2db-8890-4adc-894b-40bf0b72c1e4n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/628EAFA3-CA89-4B5F-B202-A8A11A6F95DA%40ix.netcom.com.

Re: [go-nuts] strip / reduce module compile size

[robert engels]

Something doesn’t seem right... You state the total compiled size is 22MB, but 
the first 3 entries combined are more than that.

> On Jun 14, 2024, at 4:43 PM, Tony M  wrote:
> 
> Thank you to github.com/jondot/goweight I was able to determine the compiled 
> module sizes (below). Heavy hitters are net/http & protobufs, among 100+ 
> others .  Total compile size is 22MB . 
> 
> Are there any tools or flags that can help strip or reduce some of the 
> compiled code from these modules? 
> 
>  goweight gcloud-lite
>11 MB runtime
>   7.7 MB net/http
>   5.3 MB google.golang.org/protobuf/internal/impl
>   5.2 MB google.golang.org/api/storage/v1
>   4.5 MB cloud.google.com/go/storage
>   4.2 MB golang.org/x/net/http2
>   4.0 MB golang.org/x/sys/unix
>   3.8 MB net
>   3.8 MB cloud.google.com/go/storage/internal/apiv2/storagepb
>   3.6 MB crypto/tls
>   3.4 MB google.golang.org/grpc
>   3.4 MB cloud.google.com/go/pubsub/apiv1/pubsubpb
>   3.2 MB cloud.google.com/go/pubsub/apiv1
>   2.6 MB reflect
>   2.4 MB google.golang.org/grpc/internal/transport
>   2.3 MB google.golang.org/protobuf/internal/filedesc
>   2.1 MB cloud.google.com/go/pubsub
>   1.9 MB google.golang.org/protobuf/types/descriptorpb
>   1.9 MB math/big
>   1.9 MB encoding/gob
>   1.8 MB go.opencensus.io/plugin/ochttp
>   1.7 MB syscall
>   1.7 MB cloud.google.com/go/storage/internal/apiv2
> [50+ more]
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/678b11f8-beb4-4024-91c7-1a7b18835a28n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/889A12D8-820F-42BA-A061-04ECBE81E268%40ix.netcom.com.

Re: [go-nuts] Does I/O in Goroutine blocks the underlying thread?

[robert engels]

It does. I think the wording is wrong - the Go routine blocks - but the carrier 
thread (that was running the Go routine) runs another Go routine (if needed and 
ready to run).

> On May 31, 2024, at 2:54 AM, Haoyang Fan  wrote:
> 
> Hi everyone,
> 
> I was always under the impression that Go solely uses async I/O under the 
> hood so that when invoking a seemingly blocking call like os.File.Read the 
> underlying thread won't be blocked. Go scheduler will save the context of 
> current goroutine and schedule other goroutines to run on that thread. This 
> understanding seems to be aligned with most material I can find on the 
> internet.
> 
> However, recently when I was reading the slides 
> (https://go.dev/talks/2012/waza.slide#32), on slide 32 I notice it says "When 
> a goroutine blocks, that thread blocks but no other goroutine blocks". This 
> is contradictory and make me wonder does Go really perform I/O in an 
> asynchronous manner (e.g. like select/poll/epoll in Linux) under the hood?
> 
> Can somebody please clarify?
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e1c49031-821c-42af-9449-2c7586c3676fn%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/29CDB318-EC02-4E03-A3D6-3F2498CE3E40%40ix.netcom.com.

Re: [go-nuts] Memory operations hung in semacquire state, GC routine parked while holding semaphore?

[robert engels]

I am pretty sure runtime is supposed to crash the process if it slows the 
allocators “too much” (I believe there are some config settings to control 
this).

If you have enough Go routines it may look like they are hung - you need to 
track specific routines by their ID. The stack certainly looks like it is 
trying to allocate memory for a slice - and it is being paused - because it is 
waiting on the GC lock.

Are you doing any native code? You could have heap corruption and that is also 
why the GC thread is getting hung.

If the GC thread makes progress albeit slowly - I would settle on my first 
guess - memory leak...

> On May 29, 2024, at 10:00 AM, 'Alex Kristiansen' via golang-nuts 
>  wrote:
> 
> That's an interesting idea, I probably wouldn't have thought of that on my 
> own. Is that expected behavior for memory pressure on Windows+golang? I don't 
> have much windows experience, so my assumption would be that the Windows 
> equivalent of the OOMKiller would kick in and just kill the application. 
> 
> On Tuesday, May 28, 2024 at 4:04:15 PM UTC-7 robert engels wrote:
> Feels like a memory leak to me. I would look for growing heap size in the gc 
> logs. I am guessing that the system is not completely hung - but rather the 
> runtime is having a hard time obtaining more memory, so it is slowing the 
> allocators to a rate that makes them appear hung.
> 
> It may be that the process has consumed nearly all of the OS memory too - so 
> the OS is having a hard-time responding to malloc requests.
> 
> i.e. The system is not making progress.
> 
> 
>> On May 28, 2024, at 3:54 PM, 'Alex Kristiansen' via golang-nuts 
>> > > wrote:
>> 
> 
>> This is an odd one. For reference, this is a customer machine, Windows 
>> server 2016, compiled with go1.20.11. The application just hangs after a 
>> number of days; windows minidump reveals that most threads are doing this:
>> 
>>   Goroutine 462 - User: unicode/utf16/utf16.go:106 unicode/utf16.Decode 
>> (0xe6e391) [semacquire]
>>  0  0x00e2d116 in runtime.gopark
>>  at runtime/proc.go:382
>>  1  0x00e3df5c in runtime.goparkunlock
>>  at runtime/proc.go:387
>>  2  0x00e3df5c in runtime.semacquire1
>>  at runtime/sema.go:160
>>  3  0x00e0ac2f in runtime.semacquire
>>  at runtime/sema.go:111
>>  4  0x00e0ac2f in runtime.gcMarkDone
>>  at runtime/mgc.go:787
>>  5  0x00128c10 in ???
>>  at ?:-1
>>  6  0x00dfe7da in runtime.deductAssistCredit
>>  at runtime/malloc.go:1217
>>  7  0x00dfdff0 in runtime.mallocgc
>>  at runtime/malloc.go:932
>>  8  0x00e3f972 in runtime.makeslice
>>  at runtime/slice.go:103
>>  9  0x00e6e391 in unicode/utf16.Decode
>>  at unicode/utf16/utf16.go:106
>> 10  0x00e72a7b in syscall.UTF16ToString
>>  at syscall/syscall_windows.go:63
>> 11  0x00eb7a67 in os.(*File).readdir
>>  at os/dir_windows.go:43
>> 12  0x00eb72c5 in os.(*File).Readdirnames
>>  at os/dir.go:70
>> 13  0x00fb623a in path/filepath.glob
>>  at path/filepath/match.go:346
>> 14  0x00fb5ea5 in path/filepath.globWithLimit
>>  at path/filepath/match.go:273
>> 15  0x031de255 in path/filepath.Glob
>>  at path/filepath/match.go:243
>> 
>> Multiple threads, all waiting on a semaphore inside mallocgc. The actual 
>> reason we're waiting for memory changes, obviously (sometimes a string 
>> method, or hashmap allocation, or logging call...). This behavior has 
>> remained consistent across multiple hangs across a number of weeks. After a 
>> bit of digging into the minidump, this (I think) is the thread that's 
>> holding the semaphore(s), at least based on the line numbers:
>> 
>>   Goroutine 37 - User: :0 ??? (0x7ffb0e056974) (thread 5628)
>> 0  0x7ffb0e056974 in ???
>> at ?:-1
>> 1  0x00e5c5a0 in runtime.systemstack_switch
>> at runtime/asm_amd64.s:463
>> 2  0x00e0ade5 in runtime.gcMarkDone
>> at runtime/mgc.go:855
>> 3  0x00128c10 in ???
>> at ?:-1
>> 4  0x00e5e881 in runtime.goexit
>> at runtime/asm_amd64.s:1598
>> 
>> Note that th

Re: [go-nuts] Memory operations hung in semacquire state, GC routine parked while holding semaphore?

[robert engels]

Feels like a memory leak to me. I would look for growing heap size in the gc 
logs. I am guessing that the system is not completely hung - but rather the 
runtime is having a hard time obtaining more memory, so it is slowing the 
allocators to a rate that makes them appear hung.

It may be that the process has consumed nearly all of the OS memory too - so 
the OS is having a hard-time responding to malloc requests.

i.e. The system is not making progress.

> On May 28, 2024, at 3:54 PM, 'Alex Kristiansen' via golang-nuts 
>  wrote:
> 
> This is an odd one. For reference, this is a customer machine, Windows server 
> 2016, compiled with go1.20.11. The application just hangs after a number of 
> days; windows minidump reveals that most threads are doing this:
> 
>   Goroutine 462 - User: unicode/utf16/utf16.go:106 unicode/utf16.Decode 
> (0xe6e391) [semacquire]
>  0  0x00e2d116 in runtime.gopark
>  at runtime/proc.go:382
>  1  0x00e3df5c in runtime.goparkunlock
>  at runtime/proc.go:387
>  2  0x00e3df5c in runtime.semacquire1
>  at runtime/sema.go:160
>  3  0x00e0ac2f in runtime.semacquire
>  at runtime/sema.go:111
>  4  0x00e0ac2f in runtime.gcMarkDone
>  at runtime/mgc.go:787
>  5  0x00128c10 in ???
>  at ?:-1
>  6  0x00dfe7da in runtime.deductAssistCredit
>  at runtime/malloc.go:1217
>  7  0x00dfdff0 in runtime.mallocgc
>  at runtime/malloc.go:932
>  8  0x00e3f972 in runtime.makeslice
>  at runtime/slice.go:103
>  9  0x00e6e391 in unicode/utf16.Decode
>  at unicode/utf16/utf16.go:106
> 10  0x00e72a7b in syscall.UTF16ToString
>  at syscall/syscall_windows.go:63
> 11  0x00eb7a67 in os.(*File).readdir
>  at os/dir_windows.go:43
> 12  0x00eb72c5 in os.(*File).Readdirnames
>  at os/dir.go:70
> 13  0x00fb623a in path/filepath.glob
>  at path/filepath/match.go:346
> 14  0x00fb5ea5 in path/filepath.globWithLimit
>  at path/filepath/match.go:273
> 15  0x031de255 in path/filepath.Glob
>  at path/filepath/match.go:243
> 
> Multiple threads, all waiting on a semaphore inside mallocgc. The actual 
> reason we're waiting for memory changes, obviously (sometimes a string 
> method, or hashmap allocation, or logging call...). This behavior has 
> remained consistent across multiple hangs across a number of weeks. After a 
> bit of digging into the minidump, this (I think) is the thread that's holding 
> the semaphore(s), at least based on the line numbers:
> 
>   Goroutine 37 - User: :0 ??? (0x7ffb0e056974) (thread 5628)
> 0  0x7ffb0e056974 in ???
> at ?:-1
> 1  0x00e5c5a0 in runtime.systemstack_switch
> at runtime/asm_amd64.s:463
> 2  0x00e0ade5 in runtime.gcMarkDone
> at runtime/mgc.go:855
> 3  0x00128c10 in ???
> at ?:-1
> 4  0x00e5e881 in runtime.goexit
> at runtime/asm_amd64.s:1598
> 
> Note that the precise point in gcMarkDone can change. I have another minidump 
> showing a hang at a different point in the same method:
> 
> Goroutine 19 - Go: :0 ??? (0x208351516f8) (thread 7164) [unknown wait reason 
> 30] 
> 0 0x7ffe66e66974 in ??? 
> at ?:-1 
> 1 0x00e9c5a0 in runtime.systemstack_switch 
> at runtime/asm_amd64.s:463 
> 2 0x00e4acff in runtime.gcMarkDone 
> at runtime/mgc.go:807 
> 3 0x00128c10 in ??? 
> at ?:-1 
> 4 0x00e9e881 in runtime.goexit 
> at runtime/asm_amd64.s:1598 
> 
> The commonality between these treads appears to be the `stacktrace()` method. 
> 
> Does anyone have any ideas for how to further debug this? Has anyone seen 
> anything like this? I've never seen anything like this before, and I can't 
> really reproduce it, as the only behavior is "let the application sit and run 
> for a few days." The only vaguely similar issue I've found is this, which 
> mentions interference from AV software: 
> https://github.com/golang/go/issues/52178. 
> 
> Right now I'm waiting for the results of a gctrace log, as well as setting 
> GODEBUG=asyncpreemptoff=1, but other than that, I'm a bit out of ideas. 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/9c78ef6f-0190-4981-ab1f-ed18ad2f1080n%40googlegroups.com
>  
> 

Re: [go-nuts] Errors trying to use external pkg z3

[robert engels]

If it is your own code, you have to use

import “github.com/aclements/go-z3/z3 <http://github.com/aclements/go-z3/z3>”

or you need a special go.mod file.

> On May 22, 2024, at 9:38 AM, robert engels  wrote:
> 
> What are you trying to run? z3 is a library.
> 
>> On May 22, 2024, at 9:29 AM, Kenneth Miller > <mailto:kennethadammil...@gmail.com>> wrote:
>> 
>> I did go get -u github.com/aclements/go-z3/z3 
>> <http://github.com/aclements/go-z3/z3>
>> 
>> but when I go run . I get
>> 
>> main.go:5:2: package z3 is not in std
>> 
>> the offending line is 
>> 
>> import "z3"
>> 
>> can someone help me please? I'm sure this has been asked before but I 
>> couldn't find it
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/d46b52fb-fa96-4544-914a-42bf83d75322n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/golang-nuts/d46b52fb-fa96-4544-914a-42bf83d75322n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> <mailto:golang-nuts+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/EE0FDF46-94C6-44D6-98C0-19D52C371C20%40ix.netcom.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/EE0FDF46-94C6-44D6-98C0-19D52C371C20%40ix.netcom.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/AF8256CE-EF92-432C-95E0-4F6198B3EDA2%40ix.netcom.com.

Re: [go-nuts] Errors trying to use external pkg z3

[robert engels]

What are you trying to run? z3 is a library.

> On May 22, 2024, at 9:29 AM, Kenneth Miller  
> wrote:
> 
> I did go get -u github.com/aclements/go-z3/z3
> 
> but when I go run . I get
> 
> main.go:5:2: package z3 is not in std
> 
> the offending line is 
> 
> import "z3"
> 
> can someone help me please? I'm sure this has been asked before but I 
> couldn't find it
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/d46b52fb-fa96-4544-914a-42bf83d75322n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/EE0FDF46-94C6-44D6-98C0-19D52C371C20%40ix.netcom.com.

Re: [go-nuts] Bitmask is slower than modulo calculation

[robert engels]

Hi. This is still not correct. Use the “for.. in b.N” as discussed in the blog 
in order to understand the per op difference - which will be more accurate for 
a microbenchmark timing,

But, if you really want to make a case that bit mask is slower than mod, then a 
simpler test would be better - you can do these ops on in loop using b.N to 
time the difference.



> On May 12, 2024, at 8:28 PM, Yuta MIYAKE  wrote:
> 
> Thank you. 
> 
> This is benchstat result. new test code follows:
> 
> ❯ go test -test.bench BenchmarkTestSingleModulo -count=10 -cpu=1 > modulo.txt
> ❯ go test -test.bench BenchmarkTestSingleBitMask -count=10 -cpu=1 > 
> bitmask.txt
> ❯ benchstat modulo.txt bitmask.txt
> goos: darwin
> goarch: arm64
> pkg: ringbuffer
>   │ modulo.txt │bitmask.txt│
>   │   sec/op   │   sec/opvs base   │
> TestSingleModulo6.648 ± 1%
> TestSingleBitMask6.694 ± 5%
> geomean 6.6486.694   ? ¹ ²
> 
> 
> new test code:
> 
> const BufferSize = 2 * 1024 * 1024
> 
> func benchmarkSingle(rb RingBuffer) {
>   total := 50
>   for i := 0; i < total; i++ {
>   for j := 0; j < 1000; j++ {
>   rb.Enqueue(j)
>   }
>   for j := 0; j < 1000; j++ {
>   rb.Dequeue()
>   }
>   }
> }
> 
> func BenchmarkTestSingleModulo(b *testing.B) {
>   rb := NewRingBuffer0(BufferSize)
>   b.ResetTimer()
>   benchmarkSingle(rb)
> }
> 
> func BenchmarkTestSingleBitMask(b *testing.B) {
>       rb := NewRingBuffer1(BufferSize)
>   b.ResetTimer()
>   benchmarkSingle(rb)
> }
> 
> On Monday, May 13, 2024 at 8:20:05 AM UTC+9 robert engels wrote:
> Use the Go benchmarking facilities, see 
> https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go 
> <https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go>
> 
> 
>> On May 11, 2024, at 9:57 PM, leon > > wrote:
>> 
> 
>> I'm trying to prove an optimization technique for ring buffer is effective. 
>> One of the technique is using bitmask instead of modulo to calculate a wrap 
>> around. However, in my environment, modulo is slightly faster in a test 
>> where 1 billion items are enqueued /dequeued by a single goroutine. What do 
>> you think could be the cause? 
>> 
>> Full code:
>> https://go.dev/play/p/H933oqrhPI- <https://go.dev/play/p/H933oqrhPI->
>> 
>> Environment:
>> * go version go1.21.4 darwin/arm64
>> * Apple M1 Pro
>> 
>> RingBuffer with modulo:
>> ```
>> type RingBuffer0 struct {
>>  writeIdx uint64
>>  readIdx  uint64
>>  buffers  []any
>>  size uint64
>> }
>> 
>> func NewRingBuffer0(size uint64) *RingBuffer0 {
>>  rb := &RingBuffer0{}
>>  rb.init(size)
>>  return rb
>> }
>> 
>> func (rb *RingBuffer0) init(size uint64) {
>>  rb.buffers = make([]any, size)
>>  rb.size = size
>> }
>> 
>> func (rb *RingBuffer0) Enqueue(item any) error {
>>  if rb.writeIdx-rb.readIdx == rb.size {
>>  return ErrBufferFull
>>  }
>>  rb.buffers[rb.writeIdx%rb.size] = item
>>  rb.writeIdx++
>>  return nil
>> }
>> 
>> func (rb *RingBuffer0) Dequeue() (any, error) {
>>  if rb.writeIdx == rb.readIdx {
>>  return nil, ErrBufferEmpty
>>  }
>>  item := rb.buffers[rb.readIdx%rb.size]
>>  rb.readIdx++
>>  return item, nil
>> }
>> ```
>> 
>> RingBuffer with bitmask:
>> change each module calculation to the code below
>> * rb.buffers[rb.writeIdx&(rb.size-1)] = item
>> * item := rb.buffers[rb.readIdx&(rb.size-1)]
>> 
>> Test:
>> func TestSingle(rb RingBuffer) {
>>  start := time.Now()
>>  total := 50
>>  for i := 0; i < total; i++ {
>>  for j := 0; j < 1000; j++ {
>>  rb.Enqueue(j)
>>  }
>>  for j := 0; j < 1000; j++ {
>>  rb.Dequeue()
>>  }
>>  }
>>  end := time.Now()
>>  count := total * 2000
>>  duration := end.Sub(start).Milliseconds()
>>  fmt.Printf("%d ops in %d ms\n", count, duration)
>> }
>> 
>> 
> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group a

Re: [go-nuts] Bitmask is slower than modulo calculation

[robert engels]

Use the Go benchmarking facilities, see 
https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go

> On May 11, 2024, at 9:57 PM, leon  wrote:
> 
> I'm trying to prove an optimization technique for ring buffer is effective. 
> One of the technique is using bitmask instead of modulo to calculate a wrap 
> around. However, in my environment, modulo is slightly faster in a test where 
> 1 billion items are enqueued /dequeued by a single goroutine. What do you 
> think could be the cause? 
> 
> Full code:
> https://go.dev/play/p/H933oqrhPI-
> 
> Environment:
> * go version go1.21.4 darwin/arm64
> * Apple M1 Pro
> 
> RingBuffer with modulo:
> ```
> type RingBuffer0 struct {
>   writeIdx uint64
>   readIdx  uint64
>   buffers  []any
>   size uint64
> }
> 
> func NewRingBuffer0(size uint64) *RingBuffer0 {
>   rb := &RingBuffer0{}
>   rb.init(size)
>   return rb
> }
> 
> func (rb *RingBuffer0) init(size uint64) {
>   rb.buffers = make([]any, size)
>   rb.size = size
> }
> 
> func (rb *RingBuffer0) Enqueue(item any) error {
>   if rb.writeIdx-rb.readIdx == rb.size {
>   return ErrBufferFull
>   }
>   rb.buffers[rb.writeIdx%rb.size] = item
>   rb.writeIdx++
>   return nil
> }
> 
> func (rb *RingBuffer0) Dequeue() (any, error) {
>   if rb.writeIdx == rb.readIdx {
>   return nil, ErrBufferEmpty
>   }
>   item := rb.buffers[rb.readIdx%rb.size]
>   rb.readIdx++
>   return item, nil
> }
> ```
> 
> RingBuffer with bitmask:
> change each module calculation to the code below
> * rb.buffers[rb.writeIdx&(rb.size-1)] = item
> * item := rb.buffers[rb.readIdx&(rb.size-1)]
> 
> Test:
> func TestSingle(rb RingBuffer) {
>   start := time.Now()
>   total := 50
>   for i := 0; i < total; i++ {
>   for j := 0; j < 1000; j++ {
>   rb.Enqueue(j)
>   }
>   for j := 0; j < 1000; j++ {
>   rb.Dequeue()
>   }
>   }
>   end := time.Now()
>   count := total * 2000
>   duration := end.Sub(start).Milliseconds()
>   fmt.Printf("%d ops in %d ms\n", count, duration)
> }
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/b9c4d2e0-4ab4-4d27-9359-abd8c090ae33n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/042B0E0B-459D-479D-B538-4B53FFAF55F5%40ix.netcom.com.

Re: [go-nuts] Congrats to the Go team

[Robert Engels]

Yes. It is this https://github.com/robaho/fixed/blob/5493e8737df761ffcf6be9441b4b8ae41fcf5da4/fixed_bench_test.go#L10On Apr 27, 2024, at 10:31 AM, Steven Hartland  wrote:Do you have the test code for that specific test?That would allow others to have a look at it and also confirm if the test is somehow optimising the function call away.On Sat, 27 Apr 2024 at 05:12, robert engels <reng...@ix.netcom.com> wrote:Apologies. The assembly files were reversed. But the timings remain the same.I will get the assembly of the test compiles tomorrow  - but if these differ - that suggests that the test harness between versions has changed as well - meaning can’t compare benchmark performances across versions ?Under the assumption that the register passing should be more efficient than the stack passing.On Apr 26, 2024, at 10:32 PM, Robert Engels <reng...@ix.netcom.com> wrote:Why would it be optimized away in 1.12 and not optimized away in 1.21?I could have made a mistake in my recording so I’ll test it again tomorrow. On Apr 26, 2024, at 10:03 PM, 'Keith Randall' via golang-nuts <golang-nuts@googlegroups.com> wrote:Isn't that assembly exactly the opposite? The code that is passing in registers should be 1.21, the one passing on the stack would be 1.12. At least, that's what should be the case with the register ABI that launched in 1.17 (for amd64).Please post a full program and a full command line you're using, so it is immediately obvious how one would reproduce.Also, the assembly of Fixed.Add isn't the thing I'm curious about. It is the assembly of BenchmarkAddFixed. Fixed.Add will probably be inlined into that function, and possibly optimized away.On Friday, April 26, 2024 at 5:44:01 AM UTC-7 robert engels wrote:There seems to be a material difference in the generated code.The function is:func (f Fixed) Add(f0 Fixed) Fixed {    if f.IsNaN() || f0.IsNaN() {        return NaN    }    return Fixed{fp: f.fp + f0.fp}}In 1.12 it appears to place the receiver and the argument in the AX and BX registers as a calling convention, but in 1.21 these are passed on the stack.1.12 assembly:github.com/robaho/fixed.Fixed.Add STEXT nosplit size=33 args=0x10 locals=0x0 funcid=0x0 align=0x0	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	TEXT	github.com/robaho/fixed.Fixed.Add(SB), NOSPLIT|NOFRAME|ABIInternal, $0-16	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$0, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$1, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$5, github.com/robaho/fixed.Fixed.Add.arginfo1(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$6, github.com/robaho/fixed.Fixed.Add.argliveinfo(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	PCDATA	$3, $1	0x 0 ()	NOP	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	XCHGL	AX, AX	0x0001 1 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	MOVQ	$9223372036854775807, CX	0x000b 00011 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	CMPQ	AX, CX	0x000e 00014 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	JEQ	21	0x0010 00016 ()	NOP	0x0010 00016 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	CMPQ	BX, CX	0x0013 00019 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	JNE	29	0x0015 00021 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:174)	MOVQ	github.com/robaho/fixed.NaN(SB), AX	0x001c 00028 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:174)	RET	0x001d 00029 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:176)	ADDQ	BX, AX	0x0020 00032 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:176)	RET	0x 90 48 b9 ff ff ff ff ff ff ff 7f 48 39 c8 74 05  .H.H9.t.	0x0010 48 39 cb 75 08 48 8b 05 00 00 00 00 c3 48 01 d8  H9.u.H...H..	0x0020 c3                                               .	rel 24+4 t=14 github.com/robaho/fixed.NaN+01.21.5 assembly:"".Fixed.Add STEXT nosplit size=54 args=0x18 locals=0x0	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	TEXT	"".Fixed.Add(SB), NOSPLIT|ABIInternal, $0-24	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$0, gclocals·33cdeebe80329f1fdbee7f5874cb(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$1, gclocals·33cdeebe80329f1fdbee7f5874cb(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$3, gclocals·33cdeebe80329f1fdbee7f5874cb(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	PCDATA	$2, $0	0x00

Re: [go-nuts] Congrats to the Go team

[Robert Engels]

Why would it be optimized away in 1.12 and not optimized away in 1.21?I could have made a mistake in my recording so I’ll test it again tomorrow. On Apr 26, 2024, at 10:03 PM, 'Keith Randall' via golang-nuts  wrote:Isn't that assembly exactly the opposite? The code that is passing in registers should be 1.21, the one passing on the stack would be 1.12. At least, that's what should be the case with the register ABI that launched in 1.17 (for amd64).Please post a full program and a full command line you're using, so it is immediately obvious how one would reproduce.Also, the assembly of Fixed.Add isn't the thing I'm curious about. It is the assembly of BenchmarkAddFixed. Fixed.Add will probably be inlined into that function, and possibly optimized away.On Friday, April 26, 2024 at 5:44:01 AM UTC-7 robert engels wrote:There seems to be a material difference in the generated code.The function is:func (f Fixed) Add(f0 Fixed) Fixed {    if f.IsNaN() || f0.IsNaN() {        return NaN    }    return Fixed{fp: f.fp + f0.fp}}In 1.12 it appears to place the receiver and the argument in the AX and BX registers as a calling convention, but in 1.21 these are passed on the stack.1.12 assembly:github.com/robaho/fixed.Fixed.Add STEXT nosplit size=33 args=0x10 locals=0x0 funcid=0x0 align=0x0	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	TEXT	github.com/robaho/fixed.Fixed.Add(SB), NOSPLIT|NOFRAME|ABIInternal, $0-16	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$0, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$1, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$5, github.com/robaho/fixed.Fixed.Add.arginfo1(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$6, github.com/robaho/fixed.Fixed.Add.argliveinfo(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	PCDATA	$3, $1	0x 0 ()	NOP	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	XCHGL	AX, AX	0x0001 1 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	MOVQ	$9223372036854775807, CX	0x000b 00011 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	CMPQ	AX, CX	0x000e 00014 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	JEQ	21	0x0010 00016 ()	NOP	0x0010 00016 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	CMPQ	BX, CX	0x0013 00019 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	JNE	29	0x0015 00021 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:174)	MOVQ	github.com/robaho/fixed.NaN(SB), AX	0x001c 00028 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:174)	RET	0x001d 00029 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:176)	ADDQ	BX, AX	0x0020 00032 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:176)	RET	0x 90 48 b9 ff ff ff ff ff ff ff 7f 48 39 c8 74 05  .H.H9.t.	0x0010 48 39 cb 75 08 48 8b 05 00 00 00 00 c3 48 01 d8  H9.u.H...H..	0x0020 c3                                               .	rel 24+4 t=14 github.com/robaho/fixed.NaN+01.21.5 assembly:"".Fixed.Add STEXT nosplit size=54 args=0x18 locals=0x0	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	TEXT	"".Fixed.Add(SB), NOSPLIT|ABIInternal, $0-24	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$0, gclocals·33cdeebe80329f1fdbee7f5874cb(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$1, gclocals·33cdeebe80329f1fdbee7f5874cb(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:172)	FUNCDATA	$3, gclocals·33cdeebe80329f1fdbee7f5874cb(SB)	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	PCDATA	$2, $0	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	PCDATA	$0, $0	0x 0 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	XCHGL	AX, AX	0x0001 1 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	MOVQ	"".f+8(SP), AX	0x0006 6 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	MOVQ	$9223372036854775807, CX	0x0010 00016 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:144)	CMPQ	AX, CX	0x0013 00019 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:173)	JNE	34	0x0015 00021 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:174)	MOVQ	"".NaN(SB), AX	0x001c 00028 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:174)	MOVQ	AX, "".~r1+24(SP)	0x0021 00033 (/Users/robertengels/go/src/github.com/robaho/fixed/fixed.go:174)	RET	0x0022 00034 (/Users/robertengels/go/src/gi

Re: [go-nuts] Congrats to the Go team

[Robert Engels]

I agree but in this case it is very consistent. Even if that were the case, wouldn’t that mean that 1.12 had better optimization in this case?I will dig in today and report back with the generated code. On Apr 26, 2024, at 12:17 AM, 'Keith Randall' via golang-nuts  wrote:> There is a pretty significant degradation in AddFixed() which may be concerning to the Go teamWhat is the benchmark for this?I am usually suspicious of sub-nanosecond benchmark times. Generally that indicates that the benchmark completely optimized away and all you are measuring is an empty loop.Hard to know for sure without looking at the generated code for BenchmarkAddFixed.On Thursday, April 25, 2024 at 10:54:42 AM UTC-7 Robert Engels wrote:There is a pretty significant degradation in AddFixed() which may be concerning to the Go team, because the code of AddFixed is simply:// Add adds f0 to f producing a Fixed. If either operand is NaN, NaN is returnedfunc (f Fixed) Add(f0 Fixed) Fixed {    if f.IsNaN() || f0.IsNaN() {        return NaN    }    return Fixed{fp: f.fp + f0.fp}}Here is the combined output: │ go1.12.17.txt │ go1.21.5.txt │ go1.22.2.txt │  │sec/op │sec/op  vs base   │sec/op  vs base   │ AddFixed-8 0.6000n ±  2%   0.9593n ±  1%  +59.89% (p=0.002 n=6)   0.8012n ± 12%  +33.53% (p=0.002 n=6)   AddDecimal-8   246.00n ±  1%66.47n ± 14%  -72.98% (p=0.002 n=6)66.23n ±  1%  -73.08% (p=0.002 n=6)   AddBigInt-814.400n ±  1%9.560n ±  2%  -33.61% (p=0.002 n=6)9.525n ±  7%  -33.85% (p=0.002 n=6)   AddBigFloat-8   79.90n ±  3%63.09n ±  0%  -21.03% (p=0.002 n=6)66.20n ±  1%  -17.15% (p=0.002 n=6)   MulFixed-8  4.950n ±  3%3.512n ±  0%  -29.04% (p=0.002 n=6)3.809n ±  2%  -23.06% (p=0.002 n=6)   MulDecimal-873.45n ±  3%65.90n ±  0%  -10.29% (p=0.002 n=6)67.20n ±  1%   -8.52% (p=0.002 n=6)   MulBigInt-8 17.45n ±  1%10.38n ±  2%  -40.52% (p=0.002 n=6)10.43n ±  1%  -40.23% (p=0.002 n=6)   MulBigFloat-8   36.00n ±  2%23.85n ±  1%  -33.75% (p=0.002 n=6)24.00n ±  1%  -33.35% (p=0.002 n=6)   DivFixed-8  4.700n ±  1%3.689n ±  1%  -21.51% (p=0.002 n=6)3.695n ±  2%  -21.39% (p=0.002 n=6)   DivDecimal-8767.0n ± 11%462.9n ±  0%  -39.65% (p=0.002 n=6)470.4n ±  4%  -38.68% (p=0.002 n=6)   DivBigInt-8 45.25n ±  1%34.68n ± 10%  -23.36% (p=0.002 n=6)34.98n ±  1%  -22.70% (p=0.002 n=6)   DivBigFloat-8   108.0n ±  1%110.8n ±  0%   +2.64% (p=0.002 n=6)113.6n ±  0%   +5.19% (p=0.002 n=6)   CmpFixed-8 0.3800n ±  3%   0.2500n ±  1%  -34.22% (p=0.002 n=6)   0.2511n ±  1%  -33.92% (p=0.002 n=6)   CmpDecimal-87.925n ±  1%6.942n ±  1%  -12.40% (p=0.002 n=6)6.503n ±  1%  -17.94%

Re: [go-nuts] Congrats to the Go team

['Robert Engels' via golang-nuts]

Thanks. I noticed those but didn’t look into how to address. Appreciate it. I 
will rerun.

> On Apr 25, 2024, at 12:23 PM, Steven Hartland  
> wrote:
> 
> Thanks for these, not sure if you noticed the notes from each run e.g. need 
> >= 4 samples to detect a difference at alpha level 0.05.
> 
> Basically run the benchmark with a -count=6 or more and then run the tool, 
> and you get a the comparison values which are typically the interesting bit
> 
> On Thu, 25 Apr 2024 at 00:29, Robert Engels  <mailto:rob...@me.com>> wrote:
>  │ /Users/robertengels/go1.21.5.txt │   
> /Users/robertengels/go1.22.2.txt│ 
>   
>
>  │  sec/op  │sec/op  vs base  
>   │   
>   
>  
> AddFixed-80.9603n ± ∞ ¹   0.7931n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> AddDecimal-8   66.41n ± ∞ ¹66.27n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> AddBigInt-89.452n ± ∞ ¹   10.650n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> AddBigFloat-8  63.26n ± ∞ ¹66.33n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> MulFixed-8 3.519n ± ∞ ¹3.939n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> MulDecimal-8   65.98n ± ∞ ¹67.07n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> MulBigInt-810.69n ± ∞ ¹10.49n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> MulBigFloat-8  23.72n ± ∞ ¹24.12n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> DivFixed-8 3.675n ± ∞ ¹3.661n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> DivDecimal-8   460.8n ± ∞ ¹469.6n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> DivBigInt-834.82n ± ∞ ¹34.90n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> DivBigFloat-8  110.4n ± ∞ ¹113.6n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> CmpFixed-80.2529n ± ∞ ¹   0.2784n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   

Re: [go-nuts] Congrats to the Go team

[robert engels]

gt;0 to compute geomean

 │ /Users/robertengels/go1.12.17.txt │   
/Users/robertengels/go1.22.2.txt   │
 │ allocs/op │  allocs/op   vs base 
│
AddFixed-8   0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
AddDecimal-8 8.000 ± ∞ ¹   2.000 ± ∞ ¹~ 
(p=1.000 n=1) ³
AddBigInt-8  0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
AddBigFloat-81.000 ± ∞ ¹   1.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
MulFixed-8   0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
MulDecimal-8 2.000 ± ∞ ¹   2.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
MulBigInt-8  0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
MulBigFloat-80.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
DivFixed-8   0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
DivDecimal-8 21.00 ± ∞ ¹   12.00 ± ∞ ¹~ 
(p=1.000 n=1) ³
DivBigInt-8  1.000 ± ∞ ¹   1.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
DivBigFloat-82.000 ± ∞ ¹   2.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
CmpFixed-8   0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
CmpDecimal-8 0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
CmpBigInt-8  0.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
CmpBigFloat-80.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
StringFixed-81.000 ± ∞ ¹   1.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
StringNFixed-8   1.000 ± ∞ ¹   1.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
StringDecimal-8  5.000 ± ∞ ¹   4.000 ± ∞ ¹~ 
(p=1.000 n=1) ³
StringBigInt-8   2.000 ± ∞ ¹   1.000 ± ∞ ¹~ 
(p=1.000 n=1) ³
StringBigFloat-8 8.000 ± ∞ ¹   7.000 ± ∞ ¹~ 
(p=1.000 n=1) ³
WriteTo-80.000 ± ∞ ¹   0.000 ± ∞ ¹~ 
(p=1.000 n=1) ²
geomean⁴-12.73% 
  ⁴

> On Apr 24, 2024, at 6:29 PM, 'Robert Engels' via golang-nuts 
>  wrote:
> 
>  │ /Users/robertengels/go1.21.5.txt │   
> /Users/robertengels/go1.22.2.txt│ 
>   
>
>  │  sec/op  │sec/op  vs base  
>   │   
>   
>  
> AddFixed-80.9603n ± ∞ ¹   0.7931n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> AddDecimal-8   66.41n ± ∞ ¹66.27n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> AddBigInt-89.452n ± ∞ ¹   10.650n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> AddBigFloat-8  63.26n ± ∞ ¹66.33n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> MulFixed-8 3.519n ± ∞ ¹3.939n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> MulDecimal-8   65.98n ± ∞ ¹67.07n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
>   
>
> MulBigInt-810.69n ± ∞ ¹10.49n ± ∞ ¹   ~ 
> (p=1.000 n=1) ²   
&g

Re: [go-nuts] Congrats to the Go team

['Robert Engels' via golang-nuts]

~ 
(p=1.000 n=1) ² 

 
CmpDecimal-80.000 ± ∞ ¹   0.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
CmpBigInt-8 0.000 ± ∞ ¹   0.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
CmpBigFloat-8   0.000 ± ∞ ¹   0.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
StringFixed-8   1.000 ± ∞ ¹   1.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
StringNFixed-8  1.000 ± ∞ ¹   1.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
StringDecimal-8 4.000 ± ∞ ¹   4.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
StringBigInt-8  1.000 ± ∞ ¹   1.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
StringBigFloat-87.000 ± ∞ ¹   7.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
WriteTo-8   0.000 ± ∞ ¹   0.000 ± ∞ ¹   ~ 
(p=1.000 n=1) ² 

 
geomean   ³+0.00%   
³  

> On Apr 24, 2024, at 6:20 PM, Steven Hartland  
> wrote:
> 
> What’s it look like when your run it through 
> https://pkg.go.dev/golang.org/x/perf/cmd/benchstat 
> <https://pkg.go.dev/golang.org/x/perf/cmd/benchstat> which will provide a 
> nice side by side comparison?
> 
> On Wed, 24 Apr 2024 at 19:26, 'Robert Engels' via golang-nuts 
> mailto:golang-nuts@googlegroups.com>> wrote:
> I have a fairly stable project github.com/robaho/fixed 
> <http://github.com/robaho/fixed> which is almost 100% cpu bound. It doesn’t 
> change so it makes a great way to compare the performance of different Go 
> versions using the same hardware. I took the time to re-run the tests today.
> 
> Using 1.21.17:
> 
> BenchmarkAddFixed-8 20   0.59 ns/op   
>  0 B/op  0 allocs/op  
>   
>
> BenchmarkAddDecimal-8500   243 ns/op 
> 176 B/op  8 allocs/op 
>   
> 
> BenchmarkAddBigInt-81   14.3 ns/op
>  0 B/op  0 allocs/op  
>   
>
> BenchmarkAddBigFloat-8  200078.8 ns/op
> 48 B/op  1 allocs/op  
>   
>
> BenchmarkMulFixed-8 34.88 ns/op   
>  0 B/op  0 allocs/op  
> 

Re: [go-nuts] Congrats to the Go team

[robert engels]

 

  
BenchmarkStringFixed-8  237850.64 ns/op   
24 B/op  1 allocs/op

   
BenchmarkStringNFixed-8 2393809649.66 ns/op   
24 B/op  1 allocs/op

   
BenchmarkStringDecimal-8 5196085   197.0 ns/op
56 B/op  4 allocs/op

   
BenchmarkStringBigInt-8 1030440498.00 ns/op   
16 B/op  1 allocs/op

   
BenchmarkStringBigFloat-82902165   395.2 ns/op   
176 B/op  7 allocs/op   


BenchmarkWriteTo-8  3714080531.71 ns/op   
28 B/op  0 allocs/op

> On Apr 24, 2024, at 2:52 PM, Stephen Illingworth 
>  wrote:
> 
> How does it perform with v1.22.0? I found a small but measurable drop in 
> throughput in one of my projects when compiled with 1.22.0. Issue raised here:
> 
> https://github.com/golang/go/issues/65647#issuecomment-1944830588
> 
> I have a feeling it's an issue with my older development hardware. But it's a 
> compute bound project , similar to your project, so I'd be interested in 
> hearing how you think it performs with 1.22.0
> On Wednesday 24 April 2024 at 19:27:07 UTC+1 Robert Engels wrote:
> I have a fairly stable project github.com/robaho/fixed 
> <http://github.com/robaho/fixed> which is almost 100% cpu bound. It doesn’t 
> change so it makes a great way to compare the performance of different Go 
> versions using the same hardware. I took the time to re-run the tests today.
> 
> Using 1.21.17:
> 
> BenchmarkAddFixed-8 20   0.59 ns/op   
>  0 B/op  0 allocs/op  
>   
>
> BenchmarkAddDecimal-8500   243 ns/op 
> 176 B/op  8 allocs/op 
>   
> 
> BenchmarkAddBigInt-81   14.3 ns/op
>  0 B/op  0 allocs/op  
>   
>
> BenchmarkAddBigFloat-8  200078.8 ns/op
> 48 B/op  1 allocs/op  
>   
>
> BenchmarkMulFixed-8 34.88 ns/op   
>  0 B/op  0 allocs/op  
>   
>
> BenchmarkMulDecimal-8   200072.0 ns/op
> 80 B/op  2 allocs/op  
>   
>
> BenchmarkMulBigInt-81   17.1 ns/op
>  0 B/op  0 allocs/op  
>   
>  

[go-nuts] Congrats to the Go team

['Robert Engels' via golang-nuts]

I have a fairly stable project github.com/robaho/fixed 
 which is almost 100% cpu bound. It doesn’t 
change so it makes a great way to compare the performance of different Go 
versions using the same hardware. I took the time to re-run the tests today.

Using 1.21.17:

BenchmarkAddFixed-8 20   0.59 ns/op
0 B/op  0 allocs/op 

  
BenchmarkAddDecimal-8500   243 ns/op 
176 B/op  8 allocs/op   


BenchmarkAddBigInt-81   14.3 ns/op 
0 B/op  0 allocs/op 

  
BenchmarkAddBigFloat-8  200078.8 ns/op
48 B/op  1 allocs/op

   
BenchmarkMulFixed-8 34.88 ns/op
0 B/op  0 allocs/op 

  
BenchmarkMulDecimal-8   200072.0 ns/op
80 B/op  2 allocs/op

   
BenchmarkMulBigInt-81   17.1 ns/op 
0 B/op  0 allocs/op 

  
BenchmarkMulBigFloat-8  300035.5 ns/op 
0 B/op  0 allocs/op 

  
BenchmarkDivFixed-8 34.71 ns/op
0 B/op  0 allocs/op 

  
BenchmarkDivDecimal-8200   779 ns/op 
568 B/op 21 allocs/op   


BenchmarkDivBigInt-8300046.1 ns/op 
8 B/op  1 allocs/op 

  
BenchmarkDivBigFloat-8  2000   108 ns/op  
24 B/op  2 allocs/op

   
BenchmarkCmpFixed-8 20   0.38 ns/op
0 B/op  0 allocs/op 

  
BenchmarkCmpDecimal-8   28.05 ns/op
0 B/op  0 allocs/op 

  
BenchmarkCmpBigInt-835.87 ns/op
0 B/op  0 allocs/op 

  
Be

Re: [go-nuts] Critical Section - static analysis

[robert engels]

The github link you provided results in a 404 error.

> On Apr 22, 2024, at 1:58 PM, Stephen Illingworth 
>  wrote:
> 
> Hello,
> 
> I've created a proof-of-concept for a method of defining critical sections.
> Crucially, the scheme allows for static analysis, thereby helping prevent
> violations of the critical sections.
> 
> It's a simple concept but I think it's something that can be built upon.
> 
> https://github.com/JetSetIlly/critsec
> 
> I'm posting it here because I think it's an interesting problem and I would 
> like
> to hear expert opinion. The problem is a real one but does this solution have 
> any
> merit?
> 
> -
> 
> The basic mechanism in this project is something I've used in a large rproject
> where performance is important. I wanted a way of locking access to data but
> without the risk of forgetting to unlock it. Very simple: 
> 
>   func (crit *Section) Lease(f func() error) error {
>   crit.lock.Lock()
>   defer crit.lock.Unlock()
>   return f()
>   }
>   
> I've been very happy with that but I thought it would be interesting to come 
> up
> with a method for statically analysing usage of the Lease() function. In other
> words, to detect those times when I access critical data outside of a Lease.
> 
> What's needed for this to work is a way of tagging data to say that it should
> only be accessed 'inside' a Lease().
> 
> My idea is this: define critical data by embedding a crit.Section struct in a
> new struct type:
> 
>   type Section struct {
>   lock sync.Mutex
>   }
> 
> 
>   type exampleCritSectioning struct {
>   crit.Section
>   a int
>   b bool
>   }
>   
> We can then lease the critical section:
> 
>   var C exampleCritSectioning
>   _ = C.Lease(func() error {
>   a = 100
>   return nil
>   })
> 
> 
> 
> So, critical data is identified with the crit.Section type and critical 
> sections
> by the entry and exit of the Lease function. Static analysis is now relatively
> simple:
> 
> 1. Find all instances of types that embed the crit.Section type
> 2. Find all accesses to data of those types
> 3. Check whether the callgraph to those access include a call to the Lease() 
> function
>
>
> The proof-of-concept project includes an analysis package as well as the crit
> package. A detailed example of an analysis report is given in the README
> 
> 
> Notes
> 
> I haven't used the analysistest package for testing because I had problems 
> with
> getting it to work with a go.work file. The examples/ package is used in lieue
> of that. 
> 
> A standalone driver for the analysis is in 'analysis/cmd/critcheck'
> 
> The callgraph is created with the 'x/tools/go/callgraph/vta' package. I 
> didn't put
> much though into this so there might be a better graph method. But VTA 
> provided
> the best results during experimentation
> 
> 
> Regards
> Stephen
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e2d9f1d9-958a-45df-b402-fe98a213fa67n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CB784B45-7498-4C12-98CC-07CEBCFE9DD0%40ix.netcom.com.

Re: [go-nuts] xml to json, parsing xml

[robert engels]

I don’t think that is true. There are multiple ways to model XML into json.

See this http://badgerfish.ning.com  for one of 
them.

> On Apr 23, 2024, at 11:43 AM, burak serdar  wrote:
> 
> In general, you cannot convert xml to json. They have incompatible
> models. XML elements are ordered similar to a JSON array, but in many
> cases you want to map XML elements to JSON objects, which are
> unordered name-value collections. Also, there is no JSON equivalent of
> an XML attribute.
> 
> If you want to work with XML, either use xml marshaling, or find a
> third-party DOM library.
> 
> On Tue, Apr 23, 2024 at 10:29 AM Don Caldwell  wrote:
>> 
>> Disclaimer - I am very new to golang.
>> I puzzled about this for a few days. After some struggle, I got a little 
>> program working that parses
>> arbitrary xml into a structure that json can understand. You can find it 
>> here:
>> https://github.com/dfwcnj/gxmldecode
>> 
>> On Thursday, October 7, 2021 at 10:06:30 AM UTC-4 RS wrote:
>>> 
>>> What is the best approach to convert xml to json?
>>> However I need above all to parse a very complex xml to see if related 
>>> properties are existing.
>>> I thought maybe converting it to Json maybe make it easier.
>>> 
>>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/90c0dd22-2d81-4393-b534-651a2376f386n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAMV2RqowOgbOnmGxsWegOuJ-_crQcNhsjj1Gxk3pAXhBmtNK5Q%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1415B2A9-7CB5-4D33-B47F-FB7992B7165D%40ix.netcom.com.

Re: [go-nuts] Re: Concurrent queries in a single transaction

[Robert Engels]

As you state, the driver (and db) need to support this. The synchronization for something like this is internal. But, if the operations are modifying the same tables you may end up with serialized operations anyway in order to ensure consistency while avoiding deadlocks. On Apr 20, 2024, at 12:11 AM, nilsocket  wrote:Does any solution exist for this problem?I'm facing similar problem, want to run bunch of queries inside a transaction,but want to run them concurrently, as running sequential is taking more than 1 second.On Friday, June 29, 2018 at 6:32:40 PM UTC+5:30 Space A. wrote:Hi,DB in common case will do lock/unlock when necessary for you implicitly. It is safe to call queries simultaneously. Moreover it is general approach in Enterprise solutions to have let's say hundreds of open connections sockets towards DB and execute statements from hundreds of threads. However to answer your question accurate you need to look at underlying implementation. As I understand, most of Go DB implementations don't even have connection pool capabilities. So they are even "safer" from that point because single network socket will become additional (and unnecessary) "serialization" mechanism. You could probably try to implement connection pool by yourself.Regards,среда, 27 июня 2018 г., 15:19:27 UTC+3 пользователь Ain написал:HiIt is my understanding that DB Tx is generally not safe to use concurrently, ie consider following code:    tx, err := db.Begin()    defer tx.Rollback()    ...    statements[0], err := tx.Prepare("...")    defer s1.Close()    statements[1], err := tx.Prepare("...")    defer s2.Close()    ...    wg.Add(len(statements))    for x := range statements {        go func(x int) {            defer wg.Done()            statements[x].QueryRow(...).Scan(...)        }(x)    }    wg.Wait()Here we prepare bunch of statements for a Tx which are "independent of each other", ie query different tables and scan to different variables etc. However they can't be executed concurrently, in a goroutine, as they share the same connection and thus interfere with each other?It might be safe when db driver takes care to serialise the statements but database/sql doesn't guarantee safe usage for given secenario.Is that correct?So how would you go about executing multiple queries in the same transaction context concurrenty?TIAain



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4c798067-55cd-4d20-9eba-875c4743ce73n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/B34F5737-742D-45F2-8917-8C60EC79BF6D%40ix.netcom.com.

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

Happy that it sparked an idea. I also don’t think Eli’s concern is valid. if there are other requests in flight (on different connections I assume) - let those continue - just put any new requests on a new transport for that host (after a 500 error is encountered) - then tear down the bad when it has no more requests being processed. On Apr 3, 2024, at 1:50 PM, Jim Minter  wrote:Yes, I agree, I think this approach makes sense (and should have been obvious to me as something to try...).  It could be implementable as a wrapper transport too.  I'll try it out and reply back here if it doesn't work.Thank-you!JimOn Wed, 3 Apr 2024 at 12:46, Robert Engels <reng...@ix.netcom.com> wrote:Just create a recyclable transport for the bad server and put all of the rest on a single shared transport. If one connection is returning 500 for all requests I can’t see how a different connection would solve that - unless the backend is completely broken. On Apr 3, 2024, at 7:48 AM, Eli Lindsey <e...@siliconsprawl.com> wrote:It would work, but has potentially high cost since it also causes any healthy conns in the pool to be torn down. How useful it is in practice depends on request rate, number of backends behind the lb, and ratio of healthy to unhealthy (500’ing) connections. It’s hard to tell from the description if it would work here - retrying and reusing the same busted connection could mean that the request rate is very low and there’s only one idle conn (in which case cycling the transport is a good solution), or it could mean that the unhealthy conn is quicker to respond than the pooled healthy conns and gobbles up a disproportionate share of requests.Tangential question, when the backend servers land in this state does the lb not detect and remove them?
-eli


On Apr 3, 2024, at 6:41 AM, Robert Engels <reng...@ix.netcom.com> wrote:That probably wasn’t clear. Why not create a Transport per host. Then when the 500 is encountered stop using that transport completely and create a new instance. Probably want to cancel any requests currently in flight. The connection pool is per transport. On Apr 2, 2024, at 11:05 PM, Eli Lindsey <e...@siliconsprawl.com> wrote:There isn’t a great way to handle this currently - we maintain out of tree patches to do something similar, though ours are h2 specific. The crux of the problem is that net currently lacks a usable connection pool API (there is some slightly newer discussion here, but it’s similar to the issue you linked https://github.com/golang/go/discussions/60746).If you want to stay in tree, one option may be using httptrace GotConnInfo and calling Close on the underlying connection (in direct violation of GotConnInfo’s doc). I would expect this to error out anything inflight, but otherwise be benign (though I have not checked :) ).
-eli


On Apr 2, 2024, at 3:29 PM, Jim Minter <j...@minter.uk> wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not su

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

Just create a recyclable transport for the bad server and put all of the rest on a single shared transport. If one connection is returning 500 for all requests I can’t see how a different connection would solve that - unless the backend is completely broken. On Apr 3, 2024, at 7:48 AM, Eli Lindsey  wrote:It would work, but has potentially high cost since it also causes any healthy conns in the pool to be torn down. How useful it is in practice depends on request rate, number of backends behind the lb, and ratio of healthy to unhealthy (500’ing) connections. It’s hard to tell from the description if it would work here - retrying and reusing the same busted connection could mean that the request rate is very low and there’s only one idle conn (in which case cycling the transport is a good solution), or it could mean that the unhealthy conn is quicker to respond than the pooled healthy conns and gobbles up a disproportionate share of requests.Tangential question, when the backend servers land in this state does the lb not detect and remove them?
-eli


On Apr 3, 2024, at 6:41 AM, Robert Engels  wrote:That probably wasn’t clear. Why not create a Transport per host. Then when the 500 is encountered stop using that transport completely and create a new instance. Probably want to cancel any requests currently in flight. The connection pool is per transport. On Apr 2, 2024, at 11:05 PM, Eli Lindsey  wrote:There isn’t a great way to handle this currently - we maintain out of tree patches to do something similar, though ours are h2 specific. The crux of the problem is that net currently lacks a usable connection pool API (there is some slightly newer discussion here, but it’s similar to the issue you linked https://github.com/golang/go/discussions/60746).If you want to stay in tree, one option may be using httptrace GotConnInfo and calling Close on the underlying connection (in direct violation of GotConnInfo’s doc). I would expect this to error out anything inflight, but otherwise be benign (though I have not checked :) ).
-eli


On Apr 2, 2024, at 3:29 PM, Jim Minter  wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not sure how else this could be done.  Does anyone have any thoughts?Many thanks for the help,Jim

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/34d597cf-a84c-48eb-b555-537a8768f468n%40googlegroups.com.


-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/080B6923-51DA-4DDB

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

That probably wasn’t clear. Why not create a Transport per host. Then when the 500 is encountered stop using that transport completely and create a new instance. Probably want to cancel any requests currently in flight. The connection pool is per transport. On Apr 2, 2024, at 11:05 PM, Eli Lindsey  wrote:There isn’t a great way to handle this currently - we maintain out of tree patches to do something similar, though ours are h2 specific. The crux of the problem is that net currently lacks a usable connection pool API (there is some slightly newer discussion here, but it’s similar to the issue you linked https://github.com/golang/go/discussions/60746).If you want to stay in tree, one option may be using httptrace GotConnInfo and calling Close on the underlying connection (in direct violation of GotConnInfo’s doc). I would expect this to error out anything inflight, but otherwise be benign (though I have not checked :) ).
-eli


On Apr 2, 2024, at 3:29 PM, Jim Minter  wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not sure how else this could be done.  Does anyone have any thoughts?Many thanks for the help,Jim

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/34d597cf-a84c-48eb-b555-537a8768f468n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/080B6923-51DA-4DDB-9400-B1054C1DFCE4%40siliconsprawl.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6AA93658-082F-4580-A3BD-6603D1D83394%40ix.netcom.com.

Re: [go-nuts] net/http: no Client API to close server connection based on Response #21978

[Robert Engels]

My guess is that if you are getting a 500 you have exhausted the server or capacity. So close the client completely and perform an exponential back off. You can wrap all of this at a higher level to keep the synchronous behavior. On Apr 2, 2024, at 7:37 PM, Jim Minter  wrote:That's possible, but the rate of occurrence of the issue is low (but painful when it happens), and the costs of starting a new TLS connection for every HTTP request are significant.  I'm looking for a better way.JimOn Tuesday 2 April 2024 at 15:01:30 UTC-6 Sean Liao wrote:since you already know the server is problematic, you could just set Close on the original request.On Tue, Apr 2, 2024, 15:29 Jim Minter  wrote:Hello,I was wondering if anyone had any ideas about https://github.com/golang/go/issues/21978 ("net/http: no Client API to close server connection based on Response") -- it's an old issue, but it's something that's biting me currently and I can't see a neat way to solve it.As an HTTP client, I'm hitting a case where some HTTP server instance behind a load balancer breaks and starts returning 500s (FWIW with no body) and without the "Connection: close" header.  I retry, but I end up reusing the same TCP connection to the same broken HTTP instance, so I never hit a different backend server and my retry policy is basically useless.Obviously I need to get the server owner to fix its behavior, but it would be great if, as a client, there were a way to get net/http not to reuse the connection further, in order to be less beholden to the server's behavior.This happens with both HTTP/1.1 and HTTP/2.If appropriate, I could live with the request to close the connection racing with other new requests to the same endpoint.  Getting to the point where 2 or 3 requests fail and then the connection is closed is way better than having requests fail ad infinitum.http.Transport.CloseIdleConnections() doesn't solve the problem well (a) because it's a big hammer, and (b) because there's no guarantee that the connection is idle when CloseIdleConnections() is called.FWIW I can see in `func (pc *persistConn) readLoop()` there's the following test:```goif resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable {	// Don't do keep-alive on error if either party requested a close	// or we get an unexpected informational (1xx) response.	// StatusCode 100 is already handled above.	alive = false}```I imagine that extending that to `if resp.Close || rc.req.Close || resp.StatusCode <= 199 || bodyWritable || resp.StatusCode >= 500 {` might probably help this specific case, but I imagine that's an unacceptably large behavior change for the rest of the world.I'm not sure how else this could be done.  Does anyone have any thoughts?Many thanks for the help,Jim



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/34d597cf-a84c-48eb-b555-537a8768f468n%40googlegroups.com.
- sean




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a3d208a6-90c0-42af-9a13-a8f3f0c7b21dn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1180CF7E-4287-4BD2-85A3-E91ABD1CF650%40ix.netcom.com.

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

Apologies.On Mar 9, 2024, at 1:16 PM, Ian Lance Taylor  wrote:This thread has gotten heated.  I think it's time to move on to different topics and give this discussion a rest.  Thanks.Ian



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXbXBEMCpgPH_piQH0-jZoC_Lok94Oo9%2Bbe2fDBnaWgNA%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/26C7753E-CEF8-42F9-9ED6-68513CC0BF5A%40ix.netcom.com.

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

My guess is that most applications are decoding 25gb json files with any regularity. Even transferring 25 GB over the fastest of networks takes 20 secs? So that reduces the cost to less than 10%???The test isn’t doing anything with the decoded json - maybe Go is 1000% faster in that. This is either a troll post or by someone that needs more education in performance monitoring. Sorry. On Mar 8, 2024, at 10:25 PM, Mike Schinkel  wrote:Hi Robert,Thanks for the reply.I decided to run a quick benchmark so I compared Go 1.22's json.UnMarshal(any) using an any type with PHP 8.3.2's json_decode()and found that PHP's option is 85% faster for a ~25Mb JSON file.  Optimizing it with a struct I was able to get Go down to being only about 25% slower than PHP's  json_decode().Note that they are very rough benchmarks, probably only valid in gauging magnitude, not specific performance.I wrote up the results and provided the code in this gist for anyone who might be interested.  For me, the takeaway is that if you really need fast JSON unmarshalling you probably need to look to one of the 3rd party JSON parser packages.Whether or not the extra time it takes Go to parse JSON vs. PHP is actually relevant for her use-case remains to be seen given the potential order-of-magnitude difference in web request latency.  Still, given that she seems to be controlling for that web request latency by comparing her PHP API to her Go API would indicate — sans issues like buffering or slow Go middleware — that the performance of JSON parsing made indeed by one of the reasons she is seeing a difference.Pragya, it would be really nice if you could follow up to close the loop and let us know what the actual bottleneck was and how you ended up solving it.-MikeOn Friday, March 8, 2024 at 9:58:26 PM UTC-5 Robert Engels wrote:Related, 30k of anything is largely meaningless for modern hardware in terms of cpu processing - where billions of operations a second is the norm. On Mar 8, 2024, at 8:55 PM, Robert Engels <ren...@ix.netcom.com> wrote:The point I was trying to make - maybe unclear - is that it is doubtful any raw performance differences between PHP and Go in terms of json encoding/decoding would be noticeable- unless it was a hft/hpc system which would be doubtful given that the original system was PHP. So if there is detectable difference in performance it is almost certainly related to IO management concerns (buffering, window sizes, etc)Without detailed measurements along with how they were measured it’s hard to know with any certainty- so this was a Large guess to help you know where to look. On Mar 8, 2024, at 8:33 PM, Mike Schinkel <mi...@newclarity.net> wrote:Hi Robert,I am now confused.  While your original reply to this thread made perfect sense to me, your follow up for the purposes of clarification had the opposite effect, at least for me. I am not clear the point you are trying to make.You mention the difference between compiled and interpreted and point out that typically there is not a large performance difference — where I assume you were referring to use of standard library functions in the interpreted language such as json_decode() in PHP. That is of course often true and I concur. (Of course, try writing a full JSON parser in pure PHP and we'll find a big difference in performance with a compiled language like Go.) But I am not seeing why that point is relevant in the thread because she was not asking "Why is PHP code slower than Go code?" in which case I would understand why you chose to make that distinction.You also mention that IO performance (and design(?)) over the web is typically much more relevant performance-wise, on which I also concur, but given that both her PHP and her Go API endpoints presumably had much the same web-based latency and response time concerns, I'm struggling to understand why it was relevant to state it in this context.  Of course your point simply could have been not using a buffered output stream,  but you already mentioned, and your follow up did not make that connection clear.So, can you please help me better understand the point you were trying to make?  As a follow up to my reply were you trying to imply that illustrating ways to optimize JSON parsing was something you felt I should not have posted in response to her question? That feels like in might have been your intent, but I could certainly have misinterpreted and if so would prefer to know rather than wrongly assume.-MikeOn Friday, March 8, 2024 at 8:28:38 PM UTC-5 Robert Engels wrote:Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC 

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

Related, 30k of anything is largely meaningless for modern hardware in terms of cpu processing - where billions of operations a second is the norm. On Mar 8, 2024, at 8:55 PM, Robert Engels  wrote:The point I was trying to make - maybe unclear - is that it is doubtful any raw performance differences between PHP and Go in terms of json encoding/decoding would be noticeable- unless it was a hft/hpc system which would be doubtful given that the original system was PHP. So if there is detectable difference in performance it is almost certainly related to IO management concerns (buffering, window sizes, etc)Without detailed measurements along with how they were measured it’s hard to know with any certainty- so this was a Large guess to help you know where to look. On Mar 8, 2024, at 8:33 PM, Mike Schinkel  wrote:Hi Robert,I am now confused.  While your original reply to this thread made perfect sense to me, your follow up for the purposes of clarification had the opposite effect, at least for me. I am not clear the point you are trying to make.You mention the difference between compiled and interpreted and point out that typically there is not a large performance difference — where I assume you were referring to use of standard library functions in the interpreted language such as json_decode() in PHP. That is of course often true and I concur. (Of course, try writing a full JSON parser in pure PHP and we'll find a big difference in performance with a compiled language like Go.) But I am not seeing why that point is relevant in the thread because she was not asking "Why is PHP code slower than Go code?" in which case I would understand why you chose to make that distinction.You also mention that IO performance (and design(?)) over the web is typically much more relevant performance-wise, on which I also concur, but given that both her PHP and her Go API endpoints presumably had much the same web-based latency and response time concerns, I'm struggling to understand why it was relevant to state it in this context.  Of course your point simply could have been not using a buffered output stream,  but you already mentioned, and your follow up did not make that connection clear.So, can you please help me better understand the point you were trying to make?  As a follow up to my reply were you trying to imply that illustrating ways to optimize JSON parsing was something you felt I should not have posted in response to her question? That feels like in might have been your intent, but I could certainly have misinterpreted and if so would prefer to know rather than wrongly assume.-MikeOn Friday, March 8, 2024 at 8:28:38 PM UTC-5 Robert Engels wrote:Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC systems deviate from this but by the description and reference to the previous implementation I doubt that is the case here. On Mar 8, 2024, at 6:53 PM, Mike Schinkel <mi...@newclarity.net> wrote:Hi Pragya,While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things you can look at.1. You mention you have to use a map because of response keys not being fixed. Be aware that you do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need, for example.  2. If you optionally need to parse portions of the JSON, but not always, you can use json.RawMessage to capture the properties you don't always need to parse, and then parse them only when you need to.3. You can also declare an .UnmarshalJSON() method on a struct you are passing to json.Unmarshal(), or on any struct that is a property of your top level object, and then be fully in control of parsing the data, meaning you could combine with (a) json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to.4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library.  Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole.  However, if you really need better performance it might be worth the added dependency.While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance:https://github.com/valyala/fastjsonhttps://github.com/by

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

The point I was trying to make - maybe unclear - is that it is doubtful any raw performance differences between PHP and Go in terms of json encoding/decoding would be noticeable- unless it was a hft/hpc system which would be doubtful given that the original system was PHP. So if there is detectable difference in performance it is almost certainly related to IO management concerns (buffering, window sizes, etc)Without detailed measurements along with how they were measured it’s hard to know with any certainty- so this was a Large guess to help you know where to look. On Mar 8, 2024, at 8:33 PM, Mike Schinkel  wrote:Hi Robert,I am now confused.  While your original reply to this thread made perfect sense to me, your follow up for the purposes of clarification had the opposite effect, at least for me. I am not clear the point you are trying to make.You mention the difference between compiled and interpreted and point out that typically there is not a large performance difference — where I assume you were referring to use of standard library functions in the interpreted language such as json_decode() in PHP. That is of course often true and I concur. (Of course, try writing a full JSON parser in pure PHP and we'll find a big difference in performance with a compiled language like Go.) But I am not seeing why that point is relevant in the thread because she was not asking "Why is PHP code slower than Go code?" in which case I would understand why you chose to make that distinction.You also mention that IO performance (and design(?)) over the web is typically much more relevant performance-wise, on which I also concur, but given that both her PHP and her Go API endpoints presumably had much the same web-based latency and response time concerns, I'm struggling to understand why it was relevant to state it in this context.  Of course your point simply could have been not using a buffered output stream,  but you already mentioned, and your follow up did not make that connection clear.So, can you please help me better understand the point you were trying to make?  As a follow up to my reply were you trying to imply that illustrating ways to optimize JSON parsing was something you felt I should not have posted in response to her question? That feels like in might have been your intent, but I could certainly have misinterpreted and if so would prefer to know rather than wrongly assume.-MikeOn Friday, March 8, 2024 at 8:28:38 PM UTC-5 Robert Engels wrote:Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC systems deviate from this but by the description and reference to the previous implementation I doubt that is the case here. On Mar 8, 2024, at 6:53 PM, Mike Schinkel <mi...@newclarity.net> wrote:Hi Pragya,While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things you can look at.1. You mention you have to use a map because of response keys not being fixed. Be aware that you do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need, for example.  2. If you optionally need to parse portions of the JSON, but not always, you can use json.RawMessage to capture the properties you don't always need to parse, and then parse them only when you need to.3. You can also declare an .UnmarshalJSON() method on a struct you are passing to json.Unmarshal(), or on any struct that is a property of your top level object, and then be fully in control of parsing the data, meaning you could combine with (a) json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to.4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library.  Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole.  However, if you really need better performance it might be worth the added dependency.While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance:https://github.com/valyala/fastjsonhttps://github.com/bytedance/sonichttps://github.com/buger/jsonparserhttps://github.com/mailru/easyjsonhttps://github.com/sugawarayuuta/sonnethttps://github.com/json-iterator/gohttps://github.com/segmentio/encodinghttps:

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc). Some HFT/HPC systems deviate from this but by the description and reference to the previous implementation I doubt that is the case here. On Mar 8, 2024, at 6:53 PM, Mike Schinkel  wrote:Hi Pragya,While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things you can look at.1. You mention you have to use a map because of response keys not being fixed. Be aware that you do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need, for example.  2. If you optionally need to parse portions of the JSON, but not always, you can use json.RawMessage to capture the properties you don't always need to parse, and then parse them only when you need to.3. You can also declare an .UnmarshalJSON() method on a struct you are passing to json.Unmarshal(), or on any struct that is a property of your top level object, and then be fully in control of parsing the data, meaning you could combine with (a) json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to.4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library.  Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole.  However, if you really need better performance it might be worth the added dependency.While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance:https://github.com/valyala/fastjsonhttps://github.com/bytedance/sonichttps://github.com/buger/jsonparserhttps://github.com/mailru/easyjsonhttps://github.com/sugawarayuuta/sonnethttps://github.com/json-iterator/gohttps://github.com/segmentio/encodinghttps://github.com/goccy/go-jsonhttps://github.com/simdjson/simdjsonBenchmarks from authors of some of the packages (so take with a grain of salt):https://github.com/sugawarayuuta/benchmarkhttps://blog.min.io/simdjson-go-parsing-gigabyes-of-json-per-second-in-go/And benchmarks from someone who is not an author of one of those packages:https://kokizzu.blogspot.com/2022/12/map-to-struct-and-struct-to-map-golang.html Hope this helps.-MikeOn Friday, March 8, 2024 at 4:15:25 PM UTC-5 Robert Engels wrote:It is highly unlikely that the Go marshaling is the cause. I’m guessing you are probably not using a buffered output stream. PHP is written in C but depending on what you are doing it is either a script or calling out to another process (afaik) On Mar 8, 2024, at 2:35 PM, pragya singh <pragy...@gmail.com> wrote:Hi,"I am facing an issue in my Go code. I converted my read API from PHP to Go, but the response time of Go API is higher than PHP. We are returning the response in JSON format, and the response contains around 30k keys, which is taking too much time in JSON marshaling. We are adding all the response in map format as we can't use struct because response keys are not fixed and vary from user to user.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a57a2a71-b0f9-41fb-93e9-bd54b57829b1n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7345c383-1529-489c-b156-72601bd80530n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/C19386B5-82FB-489A-896B-06807B09C6CD%40ix.netcom.com.

Re: [go-nuts] How To Reduce Json Marshalling Time

[Robert Engels]

It is highly unlikely that the Go marshaling is the cause. I’m guessing you are probably not using a buffered output stream. PHP is written in C but depending on what you are doing it is either a script or calling out to another process (afaik) On Mar 8, 2024, at 2:35 PM, pragya singh  wrote:Hi,"I am facing an issue in my Go code. I converted my read API from PHP to Go, but the response time of Go API is higher than PHP. We are returning the response in JSON format, and the response contains around 30k keys, which is taking too much time in JSON marshaling. We are adding all the response in map format as we can't use struct because response keys are not fixed and vary from user to user.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a57a2a71-b0f9-41fb-93e9-bd54b57829b1n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/32B1DA04-4F39-40B7-8A70-CB5A669248D2%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

Glad you figured it out. Not certain how requests could crash a process like… I think you’d be better off configuring a maximum heap size rather than having the OOM killer kick in On Mar 2, 2024, at 3:50 PM, Russtopia  wrote:SOLVED!Thank you all for the helpful suggestions. Although it has turned out to be something totally different, and a teachable lesson in web app design...This go tool of mine has a very simple web interface with controls for a set of jobs on the main page.The jobs on this list can be run, viewed, and most importantly, cancelled via an endpoint of the form "/cancel/?id=nnn" ...I have had the site up in a "demo" mode on the public internet at various times, including recently -- it turns out that very recently some crawl bots must have found it, and they are following those /cancel/ links on the dashboard almost as soon as they appear -- they must be scanning at <5sec intervals to find the new unique jobIDs encoded in each 'cancel' link. Oops :)The app's usually behind an auth page, but this time it wasn't. I'm not a web dev so rookie mistake I suppose!I guess I really should have a 'robots.txt' file served out by my go app to prevent this, and perhaps consider other client session ids to prevent outside crawlers from accidentally activating my app's link endpoints.Thank you again, all.On Sat, Mar 2, 2024 at 6:23 PM Robert Engels <reng...@ix.netcom.com> wrote:I would  be also try reverting the Go version and ensure that it continues to work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor <i...@golang.org> wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia <rma...@gmail.com> wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.





-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/852FA6F1-8B90-4361-B1BC-579FBEC83513%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

I’m guessing some other library or package you installed has corrupted your Linux installation. Sadly, my suggestion would be a fresh install of Linux. On Mar 2, 2024, at 1:58 PM, Russtopia  wrote:It no longer does.. so it suggests to me there's something external that has changed, but I have no clue as to what that might be -- as the process being started by my go tool will run just fine from a shell. And, it *does* run fine on my laptop (which granted is beefier, but again this server was running the tool just fine for years and I haven't done any big upgrades, but it *is* possible some minor underlying package update has severely changed the environment somehow). Unfortunately I don't have a system-wide 'snapshot' I can revert to.Perhaps this will end up being a question of Linux diagnostics more than Go but I haven't yet seen any way to tell *why* the process is being killed, whether it be due to some bug tickled by Go's exec or something else. The oom_reaper doesn't say a thing to system logs; I don't see my free RAM or swap suddenly drop.. I've even checked my server for rootkits out of paranoia :). Everything else on the system is just fine, I just cannot seem to run these scripts any more when launched from my go program (again, even a 'do-nothing' script that just sleeps a few times, then completes with exit status 0, no longer works -- it just gets 'killed').On Sat, Mar 2, 2024 at 7:39 PM Robert Engels <reng...@ix.netcom.com> wrote:Please clarify - does it work using the older versions of Go?On Mar 2, 2024, at 12:53 PM, Russtopia <rma...@gmail.com> wrote:I have tried rebuilding with go1.18.6, go1.15.15 with no difference.On Sat, Mar 2, 2024 at 6:23 PM Robert Engels <reng...@ix.netcom.com> wrote:I would  be also try reverting the Go version and ensure that it continues to work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor <i...@golang.org> wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia <rma...@gmail.com> wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.






-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/B6BF9BE3-1A33-4B05-92C2-1382B6F845CF%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

Please clarify - does it work using the older versions of Go?On Mar 2, 2024, at 12:53 PM, Russtopia  wrote:I have tried rebuilding with go1.18.6, go1.15.15 with no difference.On Sat, Mar 2, 2024 at 6:23 PM Robert Engels <reng...@ix.netcom.com> wrote:I would  be also try reverting the Go version and ensure that it continues to work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor <i...@golang.org> wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia <rma...@gmail.com> wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.





-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8E4E2802-8E23-4427-ACA1-DBA30EBD8B30%40ix.netcom.com.

Re: [go-nuts] Plea for help diagnosing strange "signal: killed" issues in previously-working code

[Robert Engels]

I would  be also try reverting the Go version and ensure that it continues to 
work. Other system libraries may have been updated. 

> On Mar 2, 2024, at 12:05 PM, Ian Lance Taylor  wrote:
> 
> On Sat, Mar 2, 2024 at 9:59 AM Russtopia  wrote:
>> 
>> Symptom: mysterious "signal: killed" occurrences with processes spawned from 
>> Go via exec.Cmd.Start()/Wait()
> 
> The first step is to tell us the exact and complete error that you
> see.  "signal: killed" can have different causes, and the rest of the
> information should help determine what is causing this one.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXR6hBSnGLjehwng%2BXp4QQ8ZznramEAZTmD%3D6tVwFirTg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/B3E066FF-18F8-4BD9-A78B-2D98B930801B%40ix.netcom.com.

Re: [go-nuts] Limiting File Descriptors on Forked Linux cmd.exec

[Robert Engels]

The could be calling fork() as in the system call - which copies all file 
descriptors but I didn’t think Go processes could fork.

Seems you would need to remap stdin and stdout in the fork to do anything 
useful. 

This sounds very PHP - what goes around comes around. 

> On Mar 1, 2024, at 8:01 PM, Ian Lance Taylor  wrote:
> 
> On Fri, Mar 1, 2024 at 5:57 PM Jeff Stein  wrote:
>> 
>> I'm struggling to understand if I'm able to do something.
>> 
>> 
>> In my very odd use case we are writing a websever that handles connections 
>> via a forked process.
>> 
>> I have a listener process that listens for TCP connections.
>> 
>> So each net.Conn that comes in we pull off its file descriptor:
>> 
>> fd, err := conn.(*net.TCPConn).File()
>> 
>> duplicate that file descriptor and then fork off a process passing in that 
>> file descriptor.
>> 
>> In my forked handler I'll reconstruct the HTTP connection and "do stuff".
>> 
>> The concern I'm having is that it appears when I fork a process I inherit 
>> all of the parent file descriptors so if I have say 5 incoming connections 
>> and then I fork my child process technically could write to a different 
>> connection.
>> 
>> I've played around with the various options:
>> 
>> cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: false,}
>> 
>> and using cmd.ExtraFiles
>> 
>> No matter what I do I seem unable to limit the sub process to ONLY using the 
>> specific File Descriptor I want it to have access to.
>> 
>> I believe this is doable in C - but I'm not sure if I can do this in GoLang 
>> as-is without mods .
> 
> What you are describing shouldn't happen.  A child process should only
> get the file descriptors explicitly passed via the os/exec.Cmd fields
> Stdin, Stdout, Stderr, and ExtraFiles.  So tell us more: OS and
> version of Go, and what is showing you that all file descriptors are
> being passed down to the child.
> 
> Ian
> 
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUb27YBCyE52QisHLyB9XPPpEycMxt4FrFJogGsFMiemQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C7B700A5-F5BE-45D2-AA8B-E8A504627408%40ix.netcom.com.

Re: [go-nuts] Could we trade all the `ctx context.Context` arguments for one pointer in `g`?

[Robert Engels]

That is a good reason as to why putting timeouts and cancellation in the “context” always felt wrong to me. These are per request notions - and creating a new context to specify them seems off. But as I mentioned in the other post, without a concept of security context it doesn’t matter much - just throw anything and everything into the context and pass it around. On Feb 21, 2024, at 4:33 AM, 'Sean Liao' via golang-nuts  wrote:https://go.dev/issue/21335- seanOn Wed, Feb 21, 2024, 00:36 Sam Vilain  wrote:

  

  
  
Alright, well thanks for your input.
I do think these questions can be answered; exploring the use
  cases in a proposal format should hopefully show that the impact
  of closures would not normally be an issue.  Possibly the worst
  case is if you had a library to some external service, and at some
  low layer you're using a closure, well you might need to refactor
  that if you wanted to add cross–service tracing support.  But to
  be honest, it's probably better that the state you're depending on
  is captured in a struct instead of being scattered about loosely
  in a scope as with a closure.  And the common practice in Go is to
  return types that satisfy interfaces, not function references that
  you blind call.

I think I will submit a proposal, but please don't take that to
  imply that I think you're "wrong, or mistaken".  Your concerns are
  legitimate and the proposal should answer them cleanly.  In the
  proposal format, hopefully the "litigation", or more exploration
  of possible uses and abuses of the system, along with performance
  concerns, can be addressed.  I don't want to be dismissive of
  them, I just want to understand them.
I had a brief look on the Golang issues in Github and could not
  find any prior proposals along this line using "context" and
  "dynamic scope" as search terms, so I'll submit this as a "new"
  proposal for now.
Thanks again, and truly—thanks for responding, >100% better
  than people who just rolled eyes and marked thread as read.
Cheers,
  Sam

On 2/20/24 3:35 PM, Axel Wagner wrote:


  
  
If I may quote myself:


> And no matter which choice you make for the language -
  it means that if the programmers wanted the other, they'd have
  to jump through annoying hoops and get confusing and hard to
  debug problems.


Having a mechanism to get one or the other semantic doesn't
  change the fact that it's easy to choose wrongly by accident,
  as long as the effect is implicit. In fact, the mechanism you
  propose (AIUI) seems extra confusing: Having a function value
  sometimes create a new dynamic scope and sometimes not, is
  weird and seems like a recipe for frustration.



But really, convincing me isn't really the point, which is
  why I'm not super invested in litigating this (otherwise I
  might try to come up with realistic examples, for instance. Or
  explain further why I'm still not sure that this can be
  implemented efficiently). I'm just re-stating what, in the
  past, where the reasons why things like this have been
  rejected. In order to predict what I would consider a likely
  outcome of a proposal like this.


If you think I am wrong or misunderstanding you, you can
  always file a proposal to get a more official response.

  
  
  
On Tue, Feb 20, 2024 at
  8:18 PM Sam Vilain 
  wrote:


  
On 2/17/24 1:32 AM, Axel Wagner wrote:


  

  On Sat, Feb 17, 2024
at 2:09 AM Sam Vilain 
wrote:
  
  
I would argue that the matter can be simply
  decided by choosing the calling stack, not
  the destination stack.
  
   
  I agree that this is *one choice*. But the point
is, that *sometimes* you'd want one and *sometimes*
the other. And no matter which choice you make for
the language - it means that if the programmers
wanted the other, they'd have to jump through
annoying hoops and get confusing and hard to debug
problems. So if you want to justify either choice,
you have to make an argument that it is so
overwhelmingly more common what people would want,
that the cost of running into these problems is

Re: [go-nuts] Could we trade all the `ctx context.Context` arguments for one pointer in `g`?

[Robert Engels]

FWIW, I think having a bound context to an execution context is a valuable addition. One thing about Go that has always felt lacking is dynamic code loading execution. There is the plugin facility but it doesn’t seem to have been widely adopted. If it were, I think the Go team would find it needs a security model closer to that of Java - and being able to bind to the thread of execution and inherit there is critical to this working well. So I welcome your proposal. On Feb 20, 2024, at 6:37 PM, Sam Vilain  wrote:

  

  
  
Alright, well thanks for your input.
I do think these questions can be answered; exploring the use
  cases in a proposal format should hopefully show that the impact
  of closures would not normally be an issue.  Possibly the worst
  case is if you had a library to some external service, and at some
  low layer you're using a closure, well you might need to refactor
  that if you wanted to add cross–service tracing support.  But to
  be honest, it's probably better that the state you're depending on
  is captured in a struct instead of being scattered about loosely
  in a scope as with a closure.  And the common practice in Go is to
  return types that satisfy interfaces, not function references that
  you blind call.

I think I will submit a proposal, but please don't take that to
  imply that I think you're "wrong, or mistaken".  Your concerns are
  legitimate and the proposal should answer them cleanly.  In the
  proposal format, hopefully the "litigation", or more exploration
  of possible uses and abuses of the system, along with performance
  concerns, can be addressed.  I don't want to be dismissive of
  them, I just want to understand them.
I had a brief look on the Golang issues in Github and could not
  find any prior proposals along this line using "context" and
  "dynamic scope" as search terms, so I'll submit this as a "new"
  proposal for now.
Thanks again, and truly—thanks for responding, >100% better
  than people who just rolled eyes and marked thread as read.
Cheers,
  Sam

On 2/20/24 3:35 PM, Axel Wagner wrote:


  
  
If I may quote myself:


> And no matter which choice you make for the language -
  it means that if the programmers wanted the other, they'd have
  to jump through annoying hoops and get confusing and hard to
  debug problems.


Having a mechanism to get one or the other semantic doesn't
  change the fact that it's easy to choose wrongly by accident,
  as long as the effect is implicit. In fact, the mechanism you
  propose (AIUI) seems extra confusing: Having a function value
  sometimes create a new dynamic scope and sometimes not, is
  weird and seems like a recipe for frustration.



But really, convincing me isn't really the point, which is
  why I'm not super invested in litigating this (otherwise I
  might try to come up with realistic examples, for instance. Or
  explain further why I'm still not sure that this can be
  implemented efficiently). I'm just re-stating what, in the
  past, where the reasons why things like this have been
  rejected. In order to predict what I would consider a likely
  outcome of a proposal like this.


If you think I am wrong or misunderstanding you, you can
  always file a proposal to get a more official response.

  
  
  
On Tue, Feb 20, 2024 at
  8:18 PM Sam Vilain 
  wrote:


  
On 2/17/24 1:32 AM, Axel Wagner wrote:


  

  On Sat, Feb 17, 2024
at 2:09 AM Sam Vilain 
wrote:
  
  
I would argue that the matter can be simply
  decided by choosing the calling stack, not
  the destination stack.
  
   
  I agree that this is *one choice*. But the point
is, that *sometimes* you'd want one and *sometimes*
the other. And no matter which choice you make for
the language - it means that if the programmers
wanted the other, they'd have to jump through
annoying hoops and get confusing and hard to debug
problems. So if you want to justify either choice,
you have to make an argument that it is so
overwhelmingly more common what people would want,
that the cost of running into these problems is
small

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

If you implemented it and it works - I guess I don’t understand your question. You must know how tcp/http works. On Feb 11, 2024, at 8:25 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:Yes I did implement. You can check it here.https://github.com/roychowdhuryrohit-dev/slugOn Sunday, February 11, 2024 at 5:00:02 PM UTC-8 Robert Engels wrote:If it is a web server that supports http clients you still have to implement the protocol correctly. At the lowest level that is what all web server implementations do - they read from the socket directly. On Feb 11, 2024, at 7:54 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:As stated earlier, objective is to create a web server without using net/http, instead directly reading from the socket connection.On Sunday, February 11, 2024 at 4:49:09 PM UTC-8 Robert Engels wrote:If you have http keep alive on - either side should block when reading - it is a full duplex connection. You will only get eof if the tcp connection is closed. If http keep alive is off, then the connection is closed after the server sends the response - the tcp protocol allows the client to read this data even though the connection is shut down. On Feb 11, 2024, at 5:36 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:Yes I got it but I want to know if reader.ReadString("\n") is supposed to throw io.EOF rather than blocking for next request in the connection.On Sunday, February 11, 2024 at 2:30:44 PM UTC-8 Robert Engels wrote:There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if i

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

If it is a web server that supports http clients you still have to implement the protocol correctly. At the lowest level that is what all web server implementations do - they read from the socket directly. On Feb 11, 2024, at 7:54 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:As stated earlier, objective is to create a web server without using net/http, instead directly reading from the socket connection.On Sunday, February 11, 2024 at 4:49:09 PM UTC-8 Robert Engels wrote:If you have http keep alive on - either side should block when reading - it is a full duplex connection. You will only get eof if the tcp connection is closed. If http keep alive is off, then the connection is closed after the server sends the response - the tcp protocol allows the client to read this data even though the connection is shut down. On Feb 11, 2024, at 5:36 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:Yes I got it but I want to know if reader.ReadString("\n") is supposed to throw io.EOF rather than blocking for next request in the connection.On Sunday, February 11, 2024 at 2:30:44 PM UTC-8 Robert Engels wrote:There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if it exists in the pool.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/919a5a2d-bd99-4f9c-b9fd-cfa2bd0a3862n%40googlegroups.

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

If you have http keep alive on - either side should block when reading - it is a full duplex connection. You will only get eof if the tcp connection is closed. If http keep alive is off, then the connection is closed after the server sends the response - the tcp protocol allows the client to read this data even though the connection is shut down. On Feb 11, 2024, at 5:36 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:Yes I got it but I want to know if reader.ReadString("\n") is supposed to throw io.EOF rather than blocking for next request in the connection.On Sunday, February 11, 2024 at 2:30:44 PM UTC-8 Robert Engels wrote:There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts <golan...@googlegroups.com> wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if it exists in the pool.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/919a5a2d-bd99-4f9c-b9fd-cfa2bd0a3862n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8014b001-ef32-4937-b31e-1ce6b24baf48n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from thi

Re: [go-nuts] Re: KeepAlive with net.Listen

[Robert Engels]

There is no such thing as a pool of idle connections at the tcp level. As each side of the connection is bound to a specific port on both ends and can’t be unbound. You may be referring to http over tcp where the client and server do not close the connection after each request - they keep it open for the next request from that client. Http2 complicates this a bit as it has multiple connections over a single tcp connection. On Feb 11, 2024, at 4:22 PM, 'Rohit Roy Chowdhury' via golang-nuts  wrote:I got your point. But reader.ReadString('\n') does not block like you said. After a request gets parsed, from the next iteration it keeps on emitting io.EOF until next request arrives.On Sunday, February 11, 2024 at 9:37:43 AM UTC-8 Brian Candler wrote:You're thinking backwards. "Long polling" is something done at the *client* side: this is where you send a HTTP request, but the reply intentionally doesn't come back for a long time - generally until the server detects some event that needs reporting.At a web *server*, you simply read the request from the socket(*), process it, reply, and go straight back to reading the next request. Read will block until the next request comes in (or the connection is closed).  In other words, the goroutine handling that TCP connection just has a loop. There's no need to "wake" this goroutine from anywhere.(*) You need to read until the end of the request (request headers + body, if any). Again, RFC2616 tells you how the request is delimited - see section 5.On Saturday 10 February 2024 at 19:12:42 UTC Rohit Roy Chowdhury wrote:Thanks, that makes so much sense. So should I long-poll until next request line comes or keep-alive times out? Is there a better way to detect incoming requests and then maybe awake the goroutine using channels?On Saturday, February 10, 2024 at 1:52:23 AM UTC-8 Brian Candler wrote:Handling keep-alives on the *server* side doesn't require any sort of connection pool. Just create one goroutine for each incoming TCP connection, and once you've handled one request, loop around, waiting for another request on the same connection.(That's assuming the client does request use of keep-alives of course; if they don't, you should close the connection. This depends on which HTTP version they requested and the Connection: header if present. Full details in RFC 2616)On Saturday 10 February 2024 at 06:08:10 UTC Rohit Roy Chowdhury wrote:Hello fellow gophers, I am currently building an experimental HTTP/1.1 framework based on TCP sockets as part of my course project. In project requirements, I have been asked to make a web server which can handle keep-alive properly without using the net/http library. The project link can be found below:https://github.com/roychowdhuryrohit-dev/slugI have recently found out that if I SetKeepAlive(true) and SetKeepAlivePeriod(time.Second * time.Duration(timeout)), it is not enough to hold the connection. Additionally, any subsequent requests are freezing.Then I found out that net/http's Transport manages a pool for idle connections. I want to go for a similar approach for my project. But I am not able to figure out how to detect income requests for my idle connections that I will be storing in the pool. Specifically, I want to know how listener.Accept() can give me an idle connection if it exists in the pool.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/919a5a2d-bd99-4f9c-b9fd-cfa2bd0a3862n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0033731F-373E-43B2-9348-11506C797D11%40ix.netcom.com.

Re: [go-nuts] help with thread limit

[Robert Engels]

You can use cpuctrl or similar to limit the number of visible cores when you start the process. On Jan 31, 2024, at 8:43 PM, Steve Roth  wrote:I am running Go code on a shared web hosting server from a major hosting company.  Using cgroups, they limit the number of threads any user can create to 25.  Up until a week ago, they had me running on a server with 24 cores, and everything worked fine.  Now they've moved me to a new server with 128 cores.  The Go runtime thinks it should be able to create that many threads, and it can't, and all of my Go programs crash with, "runtime: failed to create new OS thread".Setting GOMAXPROCS doesn't help this, since it only controls the number of active threads, not the total number of threads.  The Go runtime still thinks it can create as many threads as there are cores, and crashes the program when it's blocked from doing so.Is there any way around this?  Or is Go just completely unusable in an environment where the per-user process limit is less than the number of cores?Many thanks,Steve



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAnpqKHvfni_U%2BkCHp0hhUryDytPzQ%2BSqHZTcAFH9HNXgRCy_A%40mail.gmail.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/51EA8D38-5882-4F9B-A660-6E47627C5846%40ix.netcom.com.

Re: [go-nuts] Re: Good examples of Go back ends?

[Robert Engels]

github.com/robaho/go-trader has many easy to follow examples - authentication, multicast , routing, rest apis, websockets. A major system not covered though is db access. On Jan 22, 2024, at 9:23 AM, george looshch  wrote:hi Jason,thanks a million for pointing out the vagueness of my question! English isn’t my mother tongue so now i see where you’re coming fromwhat i meant was examples of real-world web server with routing, authentication, DB, etc.curated lists have libraries and frameworks, what i’m looking for is examples of usages these libraries and frameworks in production. Search on github didn’t yield any good results, unfortunatelyOn Monday, January 22, 2024 at 2:13:17 PM UTC Jason E. Aten wrote:This question is too vague.You are likely to get more helpful answers if you specify what kind of "backend" you are looking for.  As it is, we can only guess.  Do you want backends that are web servers? (see the standard library net/http or the caddy web server)  Is it a backend for iOS iPhone Apps? For Android Apps? That respond to a specific kind of RPC such as gRPC? That simply access a database?...  A relational database? A non-relational database (graph?, vector?, full-text search?) Pocketbase is a backend mentioned recently on hackernews, that is written in Go and seems to do alot.  Perhaps it is similar to firebase, just going by the name. I have not used it myself.  I cannot say whether it is a "good example" or not, because I've not used it, and similarly this is too vague a criteria (good at what?)https://github.com/pocketbase/pocketbaseGenerally, go over to github and search for the kind of backend you want, and select those projects that are written in Go on the left side filter click-boxes.  You could also look at the curated lists of Go libraries such as https://awesome-go.com/On Sunday, January 21, 2024 at 4:57:42 PM UTC+1 george looshch wrote:hi!can i please ask if someone knows good examples of back ends written in Go? If not good, just production code would be great as well!thanks in advance and have a great rest of the weekend!



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/bcb429bb-4cfb-4421-be4e-b1ffbcd5e228n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1A0098A5-D948-4C46-8E93-6AB01D85AA39%40ix.netcom.com.

Re: [go-nuts] Re: does anyone have an example of a Go kernel module

[Robert Engels]

Hasn’t been touched in 6 years and it doesn’t work. I guess they figured out it wasn’t worth the effort. The only plausible way to write a kernel module in Go is… don’t.Maybe run a user space Go process and some sort of shared memory communication. On Jan 7, 2024, at 8:06 PM, Hữu Hà  wrote:Maybe you mean something like this ?https://github.com/gopher-os/gopher-osVào lúc 04:22:10 UTC+7 ngày Chủ Nhật, 7 tháng 1, 2024, Siglo XIX đã viết:I have tried many ways but now that the ecosystem is more mature maybe someone knows of an example of how to make a Linux kernle module with Go,.



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cf1a03df-6782-4ef7-92af-4d3f3dc833c8n%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/28E9FBFC-4649-4FB3-8D50-5439FF5935EB%40ix.netcom.com.

Re: [go-nuts] How to convert a net.Addr into a netip.AddrPort ?

[Robert Engels]

github.com/robaho/go-trader has udp multicast support with both server and client codeOn Dec 24, 2023, at 11:05 AM, christoph...@gmail.com  wrote:Hello,I'm developping a UDP client server program. Everything is clear on the server side. I'm just a bit confused with the client side. In C we can specify the network address and set the port to 0 so that a free port is picked. Apparently we can't do that with the net API. The client must use net.DialUDP() to get a connection it can use to send messages to the remote server. If the laddr is set to nil, net.DialUDP will pick the local address and port. We thus can't pick the local address and let the system pick the port. It'll pick both. I assume it is to solve the IPv4 and IPv6 coexistence. The returned net.UDPAddr has a LocalAddress() method returning a net.Addr. How can I convert this net.Addr into a netip.AddrPort ?  I can only get a string version of the address. Are we expected to use the ResolveUDP to convert the string into the UDPAddr ?  



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f248a95d-ac11-4e9f-81d8-d53d047d24ffn%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7549AFA2-126E-4B7C-87A7-03B47F916368%40ix.netcom.com.