Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

Many thank for the perf tool, it is pretty awesome. > On 27. Aug 2019, at 13:36, Robert Engels wrote: > > Ok maybe it wasn’t the cache issue, so then try this, below a certain number > of go routines given the workload the spin and cas works, beyond a certain > point it is forced into the

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

>> >> Still, you keep ignoring this aspect - in the context of actual workloads >> the difference is negligible. >> -Original Message- >> From: changkun >> Sent: Aug 26, 2019 4:08 PM >> To: golang-nuts >> Subject: Re: [go-nuts] sync.Mutex encou

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

hint/(hint+miss) >>> 2400 697103572 17641686 0.9753175194 >>> 3200 798160789 54169784 0.9364451004 >>> 3360 1387972473 148415678 0.9033996208 >>> 3600 1824541062 272166355 0.8701934506 >>> 4000 2053779401 393586501 0.8391795437 >>> 4800 1885

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

ds > the difference is negligible. > > -Original Message- > From: changkun > Sent: Aug 26, 2019 4:08 PM > To: golang-nuts > Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when > goroutine contention more than 3400 > > Based on the pprof graph, I w

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

is negligible.-Original Message- From: changkun Sent: Aug 26, 2019 4:08 PM To: golang-nuts Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400 Based on the pprof graph, I would rather believe that the massive performance drop happens

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

>> configuration dependent - so I think I've probably reached the end of being >> able to help. >> >> And finally, it probably doesn't matter at all - if the Go routine is >> doing anything of value, 300 ns is probably an insignificant cost. >> >> >>

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

- From: changkun Sent: Aug 26, 2019 2:23 PM To: golang-nuts Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400 Your cache theory is very reasonable, but this was clear in the beginning post:  "before or after the massive increase, perfor

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

onitor the cpu cache hit/miss > ratio. > > -Original Message- > From: changkun > Sent: Aug 26, 2019 2:23 PM > To: golang-nuts > Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when > goroutine contention more than 3400 > >

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

'more work' yet constant that destroys the cache - and back into the concurrency times.-Original Message- From: changkun Sent: Aug 26, 2019 2:23 PM To: golang-nuts Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400 Your cache theory

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

You can run the process under 'perf' and monitor the cpu cache hit/miss ratio.-Original Message- From: changkun Sent: Aug 26, 2019 2:23 PM To: golang-nuts Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400 Your cache theory

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

slower than 3600 - so it is > increasing linearly with the number of Go routines. > > > -Original Message- > From: changkun > Sent: Aug 26, 2019 11:49 AM > To: golang-nuts > Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when > goroutine

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

: changkun Sent: Aug 26, 2019 11:49 AM To: golang-nuts Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400 According to your formula let's sample three points:2400 goroutines: 2.508s/(5000*2400) = 2.09 × 10^-11 s3600 goroutines: 12.219s

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

According to your formula let's sample three points: 2400 goroutines: 2.508s/(5000*2400) = 2.09 × 10^-11 s 3600 goroutines: 12.219s/(5000*3600) = 6.7883 × 10-11 seconds 4800 goroutines: 16.020s/(5000*4800) = 6.67500 × 10^-11 s One can observe that 3600 and 4800 mostly equal to

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

And it looks like the `semacquire1` executed too many `gopark`, which means indicating that `cansemacquire` failed a lot when too much contention happens. On Monday, August 26, 2019 at 6:27:14 PM UTC+2, changkun wrote: > > Sorry for late response. Do you mean the total execution was not the

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

che is exhausted - the kernel based scheduler > is more efficient - so it does suggest to me that there are some > optimizations that can be done in the Go scheduler. > > I will look at a few things this evening. > > -Original Message----- > From: changkun > Sent: Aug 21, 20

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

using L1 evictions anyway - so you never see > the optimum performance possible of the scheduler). > > -Original Message- > From: changkun > Sent: Aug 20, 2019 3:33 AM > To: golang-nuts > Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when > gor

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

, 2019 4:51 PM To: golang-nuts Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400 "less than N Go routines it fits in the L1 CPU cache," I am guessing that you are thinking of local queues on each M, the scheduler's local queue size

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

;> chan/futex version is unfair - meaning many are starved. >> >> -Original Message- >> From: changkun >> Sent: Aug 19, 2019 12:50 PM >> To: golang-nuts >> Subject: [go-nuts] sync.Mutex encounter large performance drop when >> goroutine c

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

: changkun Sent: Aug 20, 2019 3:33 AM To: golang-nuts Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400 Hi Robert,Thanks for your explanation. But how could I "logged the number of operations done per Go routine", which partic

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

many are starved. > > -Original Message- > From: changkun > Sent: Aug 19, 2019 12:50 PM > To: golang-nuts > Subject: [go-nuts] sync.Mutex encounter large performance drop when goroutine > contention more than 3400 > > I am comparing the performance regardi

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

Hi Ian Davis, I read the issue before I post this thread. I think the old issue is quite different than here. Here the problem discusses sudden performance drop and unexpected regression, but the issue#5183 only did experiment on a very limited number of goroutines, and Dmitry's answer is fair

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

On Tue, 20 Aug 2019, at 9:33 AM, changkun wrote: > Hi Robert, > > Thanks for your explanation. But how could I "logged the number of operations > done per Go routine", which particular debug settings you referring to? > It is reasonable that sync.Mutex rely on runtime scheduler but channels do

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

, you will see that the Mutex version is very fair, and the > chan/futex version is unfair - meaning many are starved. > > -Original Message- > From: changkun > Sent: Aug 19, 2019 12:50 PM > To: golang-nuts > Subject: [go-nuts] sync.Mutex encounter large performan

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

] sync.Mutex encounter large performance drop when goroutine contention more than 3400 I am comparing the performance regarding sync.Mutex and Go channels. Here is my benchmark: https://play.golang.org/p/zLjVtsSx9gdThe performance comparison visualization is as follows:What are the reasons that 1

[go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

I am comparing the performance regarding sync.Mutex and Go channels. Here is my benchmark: https://play.golang.org/p/zLjVtsSx9gd The performance comparison visualization is as follows: [image: sync.Mutex performance (1).png] What are the reasons that 1. sync.Mutex encounter a large