Re: [go-nuts] what do init functions like "math ~math" in .gox refer to?

[Xiangdong JI]

Thanks Ian.
Curious to know the purpose of those dummy functions, and for the 'import' 
functions do they have real function bodies? 

On Wednesday, August 21, 2019 at 12:40:44 PM UTC+8, Ian Lance Taylor wrote:
>
> On Tue, Aug 20, 2019 at 9:12 PM Xiangdong JI  > wrote: 
> > 
> > In 'init' section of .gox files, there might be functions like the 
> followings besides those 'import', 
> > 
> > math ~math bits ~math..z2fbits atomic 
> ~runtime..z2finternal..z2fatomic 
> > 
> > where and how are those functions generated, any source code or doc I 
> can refer to? Thanks. 
>
> These are pairs of package name and init function name.  If the init 
> function name starts with "~" then it is a dummy name and there is no 
> actual function.  Otherwise it is a real function.  The function is 
> generated by the compiler and handles initialization of package scope 
> variables that cannot be statically initialized, and calls all the 
> init functions in the package. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/322e7407-bc21-44ea-8ca3-4ad07a7851b7%40googlegroups.com.

Re: [go-nuts] what do init functions like "math ~math" in .gox refer to?

[Ian Lance Taylor]

On Tue, Aug 20, 2019 at 9:12 PM Xiangdong JI  wrote:
>
> In 'init' section of .gox files, there might be functions like the followings 
> besides those 'import',
>
> math ~math bits ~math..z2fbits atomic 
> ~runtime..z2finternal..z2fatomic
>
> where and how are those functions generated, any source code or doc I can 
> refer to? Thanks.

These are pairs of package name and init function name.  If the init
function name starts with "~" then it is a dummy name and there is no
actual function.  Otherwise it is a real function.  The function is
generated by the compiler and handles initialization of package scope
variables that cannot be statically initialized, and calls all the
init functions in the package.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXuxyqccx%3Da2Zu-Z9%3DK3MTriNt3X8rGpRVfJPpk83Q%3DWQ%40mail.gmail.com.

[go-nuts] Re: Patch RFA: [C family frontend]: avoid weird constant values in libgo/sysinfo.go

[Xiangdong JI]

Thanks a lot, Ian. the schedule is fine. 

On Wednesday, August 21, 2019 at 7:37:10 AM UTC+8, Ian Lance Taylor wrote:
>
> On Mon, Aug 12, 2019 at 8:21 PM Xiangdong JI  > wrote: 
> > 
> > The .go files generated during building gccgo seem to have a few 
> constants with weird values, for example: 
> > 
> > // sysinfo.go (on x86-64, latest gcc-9 trunk) 
> > 
> > const ___FLT128_MAX__ = 1.1 
> > const ___FLT32X_DENORM_MIN__ = 1.1 
> > 
> > as a comparison, gollvm generates expected values. 
> > Could it be caused by incorrect building setting? 
>
> Per later discussion, this problem is fixed by this patch. 
> Bootstrapped and ran full testsuite on x86_64-pc-linux-gnu. 
>
> OK for mainline? 
>
> Ian 
>
>
> 2019-08-20  Ian Lance Taylor  > 
>
> * c-cppbuiltin.c (builtin_define_with_hex_fp_value): Always expand 
> when using -fgo-dump-spec. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3ad9946e-6a97-469c-9adf-5b8fdadfd677%40googlegroups.com.

[go-nuts] what do init functions like "math ~math" in .gox refer to?

[Xiangdong JI]

Hello,

In 'init' section of .gox files, there might be functions like the 
followings besides those 'import',

math ~math bits ~math..z2fbits atomic 
~runtime..z2finternal..z2fatomic  

where and how are those functions generated, any source code or doc I can 
refer to? Thanks. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/97cedf72-39bb-4875-ab03-0ef6e92cde44%40googlegroups.com.

Re: [go-nuts] Existing production-ready libraries for parallelizing HTTP requests?

[Sam Fourman Jr.]

I am also in the same boat as tom, there is certainly a demand for this
type of library.

-- Sam Fourman

On Tue, Aug 20, 2019 at 3:33 PM 'Thomas Bushnell, BSG' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> I am of the opinion that a case like this is best handled by simply
> writing the thing you want.
>
> Concurrency limits are easily managed by using tokens to gate fetches. One
> simple technique is to make a channel of struct{} with capacity equal to
> the maximum number of concurrent connections you are allowed. You can
> either fill it with things at startup, and then read from it before a
> request and send back to the channel when done, or start with it empty and
> send to it before a request and read back after. Either is equivalent.
>
> I'm not sure the point of overall concurrency limits in general, but the
> same thing works. It's unlikely to be a problem IMO for the size of job you
> describe.
>
> Retries are best done inside each fetch; wrap http.Get with the logic you
> want. There is no one-size-fits all here. There is a public backoff library
> available, but it's a bit complex and the code could easily be simpler if
> you address exactly what you want directly.
>
> For contexts, just use the http package's (*Request).WithContext method.
> That accomplishes timeouts too.
>
> On Mon, Aug 19, 2019 at 12:20 PM tom via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> tl;dr Do you of any libraries for parallelizing HTTP requests with
>> per-server concurrency control and handling of retries?
>>
>> I'm writing a service that fetches many independent small binary blobs
>> (map tiles) over HTTP from a several upstream servers and package them
>> together in to a single archive. I want to parallelize the fetching of the
>> small binary blobs. Currently there are O(10) upstream servers and O(1000)
>> small binary blobs fetched from each.
>>
>> Making parallel HTTP requests in Go is trivially easy and is demonstrated
>> in many Go tutorials and blog posts. However, I'm looking for a "production
>> ready" library that supports:
>> * Per upstream server concurrency limits.
>> * Overall (across all upstream servers) concurrency limits.
>> * Controllable retries with exponential backoff in the case of upstream
>> server errors.
>> * Timeouts for upstream requests.
>> * context.Context support.
>>
>> This would seem to be a common enough task that I would expect to find an
>> existing library that does all of the above. Existing Go web scrapers, e.g.
>> colly , likely have this functionality internally
>> but do not expose it in their API and are instead focused on crawling web
>> pages.
>>
>> Do you know of any such library?
>>
>> Many thanks,
>> Tom
>>
>> Confidentiality Notice:
>> This electronic message and any attached documents contain confidential
>> and privileged information and is for the sole use of the individual or
>> entity to whom it is addressed. If you are not the addressee of this email,
>> or the employee or agent responsible for delivering it to the addressee,
>> you are hereby notified that any dissemination, distribution or copying of
>> this transmission is strictly prohibited. If you receive this message in
>> error, please notify the sender immediately by return e-mail or telephone
>> and destroy the attached message (and all attached documents) immediately.
>> Thank you for your cooperation.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/cfb138b2-88d4-46ee-9315-996389718bad%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CA%2BYjuxvjvAQU-R7PAeocEJ%3Dp9-k0MSqRL%2BtcL4XhXaXjV%3DUepw%40mail.gmail.com
> 
> .
>


-- 

Sam Fourman Jr.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOFF%2BZ0hpyrJ8aHV0QWVC4j8BiYQKZ2RFwbkBhrBobMLESYgxg%40mail.gmail.com.

[go-nuts] Re: How to convert reflect.Value to time.Time

[Moises Soto]

Well, this is really old but I had the same problem and its because you 
need to export the field on YOUR struct (time.Time doesn't needs to have 
exported fields like someone else said).

Here's the same code with element *t time.Time* declared as *T time.Time: *
https://play.golang.org/p/tXh2LqT8s7S

On Wednesday, February 12, 2014 at 2:03:14 AM UTC-4, Go newbie wrote:
>
> Hi All,
>
> http://play.golang.org/p/ux3iiizTRx
>
> //panic: reflect.Value.Interface: cannot return value obtained from 
> unexported field or method
>
> Thanks in advance.
>
> Best Regards
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/86bb3a9e-ff60-4c2e-8d49-95ffc4f60adf%40googlegroups.com.

Re: [go-nuts] How to build gollvm on arm platform

[eric fang]

Hi Than,

I got it, thanks for your detailed explanation. And I'm also thinking about 
how to write test cases, the cabi-testgen project would be a great help, I 
will keep you informed if there are other questions or updates, thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/56ba9fab-0b7b-42a5-84f6-c88d045bffd5%40googlegroups.com.

[go-nuts] Re: Running tests per package with Go module

[Dr Waryaa]

Hi Jacques,
Thanks for trying that. I found out that I was on a wild goose chase. As it 
turns out, it was code that my TestMain
called which had a hard-coded reference looking for the GOPATH.

Cheers,
Chris

On Wednesday, August 21, 2019 at 5:43:51 AM UTC+10, Jacques Supcik wrote:
>
> Hello Chris,
>
> I made a small project with the same structure as yours to reproduce the 
> problem : https://github.com/supcik/rtppwgm
>
> But in my project, everything works as expected :
>
> » go test -count=1 ./...
> ok  demo/pkg1   0.006s
> ok  demo/pkg2   0.006s
>
> » go test -count=1 ./pkg1/...
> ok  demo/pkg1   0.006s
>
> » go test -count=1 ./pkg2/...
> ok  demo/pkg2   0.005s
>
> Can you make a similar project that exposes your issue ?
>
> Cheers,
>
> -- Jacques
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c57ec25a-9559-4365-a866-16aee4efa15b%40googlegroups.com.

[go-nuts] asm: FUNCDATA PCDATA

[Gert]

https://golang.org/doc/asm

$ cat x.go
package main

func main() {
println(3)
}
$ GOOS=linux GOARCH=amd64 go tool compile -S x.go# or: go build 
-gcflags -S x.go

--- prog list "main" ---
 (x.go:3) TEXTmain+0(SB),$8-0
0001 (x.go:3) FUNCDATA $0,gcargs·0+0(SB)
0002 (x.go:3) FUNCDATA $1,gclocals·0+0(SB)
0003 (x.go:4) MOVQ$3,(SP)
0004 (x.go:4) PCDATA  $0,$8
0005 (x.go:4) CALL,runtime.printint+0(SB)
0006 (x.go:4) PCDATA  $0,$-1
0007 (x.go:4) PCDATA  $0,$0
0008 (x.go:4) CALL,runtime.printnl+0(SB)
0009 (x.go:4) PCDATA  $0,$-1
0010 (x.go:5) RET ,
...


The FUNCDATA and PCDATA directives contain information for use by the 
garbage collector; they are introduced by the compiler.


1) funcdata What is this .0 which is not a dot :)

2) pcdata what does $-1 mean here

3) Is there some more information on funcdata and pcdata? Doesn't have to 
be in detail, because i know its a can of worms, but just enough so I can 
tell what the garbage collector will be doing :)


--


When defining a TEXT, specifying frame size $-4 tells the linker that this 
is a leaf function that does not need to save LR on entry.


4) What is LR?

5) Leaf as in closure function?


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/34c742e7-a594-4985-adfe-38c20757099c%40googlegroups.com.

Re: [go-nuts] net library

[Ian Lance Taylor]

On Mon, Aug 19, 2019 at 6:44 PM joe mcguckin
 wrote:
>
> I was perusing the NET library web page today.
>
> 1) Are the packages listed in any particular order?

The types and functions are listed alphabetically.

> 2) What is the hierarchy? e.g., which functions correspond to the low level 
> Unix network calls,
>  then there must be some higher level convenience functions. Is there a 
> list of how the functions are ordered?

The overall package description at the top of
https://golang.org/pkg/net is intended to describe the higher level
functions, which are Dial and Listen.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcV1wRZR1cqf_reV%3D%3Dgp_WMiAEpvX2Heu%2B%3DBzdUdkjDf_g%40mail.gmail.com.

[go-nuts] Empty cpu profile file?

[joe mcguckin]

Using Dave Cheney's profile package to figure out the execution path of a 
library I'm trying to understand.

The application starts, (it's a server), I hit ^c after a couple of 
minutes, and the console says

Stopping Profiling
Stopping

Main() catches the console interrupt:
signal.Notify(c, os.Interrupt)
log.Info("Stopping")

But the cpu profile file is empty. Is there a way to force is to sync 
before exiting?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b46a2b7f-0ac7-4e5a-a1e2-268a31f3afe8%40googlegroups.com.

Re: [go-nuts] net library

[joe mcguckin]

https://golang.org/pkg/net/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/21bcd226-8829-43cf-a6ca-6490d1f248c4%40googlegroups.com.

[go-nuts] Patch RFA: [C family frontend]: avoid weird constant values in libgo/sysinfo.go

[Ian Lance Taylor]

On Mon, Aug 12, 2019 at 8:21 PM Xiangdong JI  wrote:
>
> The .go files generated during building gccgo seem to have a few constants 
> with weird values, for example:
>
> // sysinfo.go (on x86-64, latest gcc-9 trunk)
>
> const ___FLT128_MAX__ = 1.1
> const ___FLT32X_DENORM_MIN__ = 1.1
>
> as a comparison, gollvm generates expected values.
> Could it be caused by incorrect building setting?

Per later discussion, this problem is fixed by this patch.
Bootstrapped and ran full testsuite on x86_64-pc-linux-gnu.

OK for mainline?

Ian


2019-08-20  Ian Lance Taylor  

* c-cppbuiltin.c (builtin_define_with_hex_fp_value): Always expand
when using -fgo-dump-spec.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXVR44KamVoYm7W2g4-Y7ibD1MPOiLvhe%3Dr-es1Ay%3DNXA%40mail.gmail.com.
Index: c-family/c-cppbuiltin.c
===
--- c-family/c-cppbuiltin.c (revision 274749)
+++ c-family/c-cppbuiltin.c (working copy)
@@ -1643,6 +1643,7 @@ builtin_define_with_hex_fp_value (const
   /* This is very expensive, so if possible expand them lazily.  */
   if (lazy_hex_fp_value_count < LAZY_HEX_FP_VALUES_CNT
   && flag_dump_macros == 0
+  && flag_dump_go_spec == NULL
   && !cpp_get_options (parse_in)->traditional)
 {
   if (lazy_hex_fp_value_count == 0)

Re: [go-nuts] Re: Running tests per package with Go module

[Marcin Romaszewicz]

It could fail if you're under $GOPATH and GO111MODULE=auto, since inside
the GOPATH, auto=off

-- Marcin


On Tue, Aug 20, 2019 at 12:43 PM Jacques Supcik  wrote:

> Hello Chris,
>
> I made a small project with the same structure as yours to reproduce the
> problem : https://github.com/supcik/rtppwgm
>
> But in my project, everything works as expected :
>
> » go test -count=1 ./...
> ok  demo/pkg1   0.006s
> ok  demo/pkg2   0.006s
>
> » go test -count=1 ./pkg1/...
> ok  demo/pkg1   0.006s
>
> » go test -count=1 ./pkg2/...
> ok  demo/pkg2   0.005s
>
> Can you make a similar project that exposes your issue ?
>
> Cheers,
>
> -- Jacques
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/22e3a0fc-ed57-4b53-9ad5-faefab58c800%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LsU%3DfjvmxSGtGe7qjN_t1xsvvF8BFVgYeJN5cfqLTO6%3Dg%40mail.gmail.com.

[go-nuts] Re: Running tests per package with Go module

[Jacques Supcik]

Hello Chris,

I made a small project with the same structure as yours to reproduce the 
problem : https://github.com/supcik/rtppwgm

But in my project, everything works as expected :

» go test -count=1 ./...
ok  demo/pkg1   0.006s
ok  demo/pkg2   0.006s

» go test -count=1 ./pkg1/...
ok  demo/pkg1   0.006s

» go test -count=1 ./pkg2/...
ok  demo/pkg2   0.005s

Can you make a similar project that exposes your issue ?

Cheers,

-- Jacques

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/22e3a0fc-ed57-4b53-9ad5-faefab58c800%40googlegroups.com.

Re: [go-nuts] Existing production-ready libraries for parallelizing HTTP requests?

['Thomas Bushnell, BSG' via golang-nuts]

I am of the opinion that a case like this is best handled by simply writing
the thing you want.

Concurrency limits are easily managed by using tokens to gate fetches. One
simple technique is to make a channel of struct{} with capacity equal to
the maximum number of concurrent connections you are allowed. You can
either fill it with things at startup, and then read from it before a
request and send back to the channel when done, or start with it empty and
send to it before a request and read back after. Either is equivalent.

I'm not sure the point of overall concurrency limits in general, but the
same thing works. It's unlikely to be a problem IMO for the size of job you
describe.

Retries are best done inside each fetch; wrap http.Get with the logic you
want. There is no one-size-fits all here. There is a public backoff library
available, but it's a bit complex and the code could easily be simpler if
you address exactly what you want directly.

For contexts, just use the http package's (*Request).WithContext method.
That accomplishes timeouts too.

On Mon, Aug 19, 2019 at 12:20 PM tom via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> tl;dr Do you of any libraries for parallelizing HTTP requests with
> per-server concurrency control and handling of retries?
>
> I'm writing a service that fetches many independent small binary blobs
> (map tiles) over HTTP from a several upstream servers and package them
> together in to a single archive. I want to parallelize the fetching of the
> small binary blobs. Currently there are O(10) upstream servers and O(1000)
> small binary blobs fetched from each.
>
> Making parallel HTTP requests in Go is trivially easy and is demonstrated
> in many Go tutorials and blog posts. However, I'm looking for a "production
> ready" library that supports:
> * Per upstream server concurrency limits.
> * Overall (across all upstream servers) concurrency limits.
> * Controllable retries with exponential backoff in the case of upstream
> server errors.
> * Timeouts for upstream requests.
> * context.Context support.
>
> This would seem to be a common enough task that I would expect to find an
> existing library that does all of the above. Existing Go web scrapers, e.g.
> colly , likely have this functionality internally
> but do not expose it in their API and are instead focused on crawling web
> pages.
>
> Do you know of any such library?
>
> Many thanks,
> Tom
>
> Confidentiality Notice:
> This electronic message and any attached documents contain confidential
> and privileged information and is for the sole use of the individual or
> entity to whom it is addressed. If you are not the addressee of this email,
> or the employee or agent responsible for delivering it to the addressee,
> you are hereby notified that any dissemination, distribution or copying of
> this transmission is strictly prohibited. If you receive this message in
> error, please notify the sender immediately by return e-mail or telephone
> and destroy the attached message (and all attached documents) immediately.
> Thank you for your cooperation.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/cfb138b2-88d4-46ee-9315-996389718bad%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BYjuxvjvAQU-R7PAeocEJ%3Dp9-k0MSqRL%2BtcL4XhXaXjV%3DUepw%40mail.gmail.com.

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

[Robert Engels]

I am assuming that there is an internal Go structure/process that when there is less than N Go routines it fits in the L1 CPU cache, and beyond a certain point it spills to the L2 or higher - thus the nearly order of magnitude performance decrease, yet consistent times within a range.Since the worker code is so trivial, you are seeing this. Most worker code is not as trivial so the overhead of the locking/scheduler constructs have far less effect (or the worker is causing L1 evictions anyway - so you never see the optimum performance possible of the scheduler).-Original Message-
From: changkun 
Sent: Aug 20, 2019 3:33 AM
To: golang-nuts 
Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

Hi Robert,Thanks for your explanation. But how could I "logged the number of operations done per Go routine", which particular debug settings you referring to?It is reasonable that sync.Mutex rely on runtime scheduler but channels do not. However, it is unclear why a significant performance drop appears. Is it possible to determine when the performance will appear?Best,ChangkunOn Monday, August 19, 2019 at 10:27:19 PM UTC+2, Robert Engels wrote:I think you'll find the reason that the Mutex uses the Go scheduler. The chan is controlled by a 'mutex' which eventually defers to the OS futex - and the OS futex is probably more efficient at scheduling in the face of large contention - although you would think it should be the other way around.I am guessing that if you logged the number of operations done per Go routine, you will see that the Mutex version is very fair, and the chan/futex version is unfair - meaning many are starved.-Original Message-
From: changkun 
Sent: Aug 19, 2019 12:50 PM
To: golang-nuts 
Subject: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

I am comparing the performance regarding sync.Mutex and Go channels. Here is my benchmark: https://play.golang.org/p/zLjVtsSx9gdThe performance comparison visualization is as follows:What are the reasons that 1. sync.Mutex encounter a large performance drop when the number of goroutines goes higher than roughly 3400?2. Go channels are pretty stable but slower than sync.Mutex before?Raw bench data by benchstat (go test -bench=. -count=5):MutexWrite/goroutines-2400-8  48.6ns ± 1%MutexWrite/goroutines-2480-8  49.1ns ± 0%MutexWrite/goroutines-2560-8  49.7ns ± 1%MutexWrite/goroutines-2640-8  50.5ns ± 3%MutexWrite/goroutines-2720-8  50.9ns ± 2%MutexWrite/goroutines-2800-8  51.8ns ± 3%MutexWrite/goroutines-2880-8  52.5ns ± 2%MutexWrite/goroutines-2960-8  54.1ns ± 4%MutexWrite/goroutines-3040-8  54.5ns ± 2%MutexWrite/goroutines-3120-8  56.1ns ± 3%MutexWrite/goroutines-3200-8  63.2ns ± 5%MutexWrite/goroutines-3280-8  77.5ns ± 6%MutexWrite/goroutines-3360-8   141ns ± 6%MutexWrite/goroutines-3440-8   239ns ± 8%MutexWrite/goroutines-3520-8   248ns ± 3%MutexWrite/goroutines-3600-8   254ns ± 2%MutexWrite/goroutines-3680-8   256ns ± 1%MutexWrite/goroutines-3760-8   261ns ± 2%MutexWrite/goroutines-3840-8   266ns ± 3%MutexWrite/goroutines-3920-8   276ns ± 3%MutexWrite/goroutines-4000-8   278ns ± 3%MutexWrite/goroutines-4080-8   286ns ± 5%MutexWrite/goroutines-4160-8   293ns ± 4%MutexWrite/goroutines-4240-8   295ns ± 2%MutexWrite/goroutines-4320-8   280ns ± 8%MutexWrite/goroutines-4400-8   294ns ± 9%MutexWrite/goroutines-4480-8   285ns ±10%MutexWrite/goroutines-4560-8   290ns ± 8%MutexWrite/goroutines-4640-8   271ns ± 3%MutexWrite/goroutines-4720-8   271ns ± 4%ChanWrite/goroutines-2400-8  158ns ± 3%ChanWrite/goroutines-2480-8  159ns ± 2%ChanWrite/goroutines-2560-8  161ns ± 2%ChanWrite/goroutines-2640-8  161ns ± 1%ChanWrite/goroutines-2720-8  163ns ± 1%ChanWrite/goroutines-2800-8  166ns ± 3%ChanWrite/goroutines-2880-8  168ns ± 1%ChanWrite/goroutines-2960-8  176ns ± 4%ChanWrite/goroutines-3040-8  176ns ± 2%ChanWrite/goroutines-3120-8  180ns ± 1%ChanWrite/goroutines-3200-8  180ns ± 1%ChanWrite/goroutines-3280-8  181ns ± 2%ChanWrite/goroutines-3360-8  183ns ± 2%ChanWrite/goroutines-3440-8  188ns ± 3%ChanWrite/goroutines-3520-8  190ns ± 2%ChanWrite/goroutines-3600-8  193ns ± 2%ChanWrite/goroutines-3680-8  196ns ± 3%ChanWrite/goroutines-3760-8  199ns ± 2%ChanWrite/goroutines-3840-8  206ns ± 2%ChanWrite/goroutines-3920-8  209ns ± 2%ChanWrite/goroutines-4000-8  206ns ± 2%ChanWrite/goroutines-4080-8  209ns ± 2%ChanWrite/goroutines-4160-8  208ns ± 2%ChanWrite/goroutines-4240-8  209ns ± 3%ChanWrite/goroutines-4320-8  213ns ± 2%ChanWrite/goroutines-4400-8  209ns ± 2%ChanWrite/goroutines-4480-8  211ns ± 1%ChanWrite/goroutines-4560-8  213ns ± 2%ChanWrite/goroutines-4640-8  215ns ± 1%ChanWrite/goroutines-4720-8  218ns ± 3%



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit 

Re: [go-nuts] How to build gollvm on arm platform

['Than McIntosh' via golang-nuts]

Sent https://go-review.googlesource.com/c/gollvm/+/190900 to improve
comments.
Thanks, Than


On Tue, Aug 20, 2019 at 9:33 AM Than McIntosh  wrote:

> Hi Eric,
>
> Thanks for the note. Your question about the code in
> CABIOracle::canPassDirectly, e.g .
>
>   if (regsInt + regsSSE == 1)
> return true;
>
> is a good one.  I'll see about adding some comments to the code there.
>
> By way of explanation, the Gollvm code is modeled after the approach used
> in clang, which does something similar in this case. For example, consider
> the following small C program:
>
> typedef unsigned long long u;
> extern u bar(u p1, u p2, u p3, u p4, u p5, u p6, u p7, u p8, u p9, u p10);
> u foo(u p) {
>   return bar(1,2,3,4,5,6,7,8,9,p);
> }
>
> Let's say you compile this code with clang for x86_64 and look at the
> emitted LLVM IR and emitted assembly, e.g.
>
> clang -c -S file.c -O
> clang -c -S file.c -O -emit-llvm
>
> In the *.ll file you'll see this LLVM IR for 'foo':
>
> define i64 @foo(i64) local_unnamed_addr #0 {
>   %2 = tail call i64 @bar(i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64
> %0) #2
>   ret i64 %2
> }
>
> What is interesting here is that the function being called, "bar", has 7
> arguments. Under the rules of the C/C++ ABI for x86_64, not all of the
> parameters can be passed in registers -- some of them have to go in memory.
> However looking at the signature, there are no memory arguments, everything
> is being passed by value. Now let's look at the assembly:
>
> foo:
> pushq %rax
> *movq %rdi, (%rsp)*
> movl $1, %edi
> movl $2, %esi
> movl $3, %edx
> movl $4, %ecx
> movl $5, %r8d
> movl $6, %r9d
> callq bar
> popq %rcx
> retq
>
>
> As you can see, the back end knows about the rules of the C/C++ ABI for
> x86_64, and it has put the right things in registers and in memory so that
> the ABI is honored.
>
> So essentially what's happening is that the front end (clang) is playing a
> little game with the back end -- for the more complicated cases involving
> aggregates and things bigger than a single register, the front end makes
> the LLVM IR explicit with regard to what gets passed in memory via by
> value. For small parameters (things that will fit into a single register)
> it just leaves them by value and counts on having the back end sort it out.
> Doing things this way has advantages from a performance perspective.
>
> For your second question, yes, your approach seems reasonable --
> making CABIOracle into a base class and then adding virtual methods seems
> like the right way to go. I don't have any very specific advice on classes
> like CABIParamInfo and EightByteInfo -- I would say the former seems a bit
> more abstract and reusable whereas the notion of an "EightByte" seems
> pretty specific to the language in the x86_64 abi description. I would just
> take your best shot and see how things go.
>
> Hope this helps, let me know if you have other questions...
>
> Thanks, Than
>
>
>
>
> On Tue, Aug 20, 2019 at 8:42 AM eric fang  wrote:
>
>> Hi Than,
>>
>> I'm trying to implement the abi part for arm64, and got two questions for
>> you:
>>
>> 1, function CABIOracle::canPassDirectly in
>> gollvm/bridge/go-llvm-cabi-oracle.cpp, I'm not quite understand the first
>> if statement:
>>   if (regsInt + regsSSE == 1)
>>return true;
>>   Why not consider whether there are enough available registers here?
>>
>> 2, I plan to abstract the CABIOracle class into a base class, and each
>> architecture implements an inheritance class based on this class. But there
>> are some auxiliary classes and structures (like class CABIParamInfo and
>> EightByteInfo etc.) in this file that are also x86 specific. I haven't
>> figured out how to handle these yet, maybe some of them are also reusable.
>> Do you think this design is reasonable? Any suggestions? I should submit a
>> prototype patch, which would be more convenient to discuss, but I have not
>> implemented it yet. . .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/702e3129-b72a-411d-80fe-70e9daae207d%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BUr55H-iUvayr4iREbK%2BgqWs%2Bt9h1fBNSpqzMujSnLke7VpNw%40mail.gmail.com.

Re: [go-nuts] How to build gollvm on arm platform

['Than McIntosh' via golang-nuts]

Forgot to add:

When I was writing the x86_64 ABI code for gollvm, I spent some time
creating an ABI test harness to find problems in my code. You can find it
at:

https://github.com/thanm/cabi-testgen

The overall idea is that it randomly generates a lot of parameter-passing
code, then you can build half of the code with a "known good" compiler (in
this case gccgo would be the logical choice) and then the other half with
the compiler you are developing.  This may be useful once you get a bit
farther along.

Cheers, Than


On Tue, Aug 20, 2019 at 9:33 AM Than McIntosh  wrote:

> Hi Eric,
>
> Thanks for the note. Your question about the code in
> CABIOracle::canPassDirectly, e.g .
>
>   if (regsInt + regsSSE == 1)
> return true;
>
> is a good one.  I'll see about adding some comments to the code there.
>
> By way of explanation, the Gollvm code is modeled after the approach used
> in clang, which does something similar in this case. For example, consider
> the following small C program:
>
> typedef unsigned long long u;
> extern u bar(u p1, u p2, u p3, u p4, u p5, u p6, u p7, u p8, u p9, u p10);
> u foo(u p) {
>   return bar(1,2,3,4,5,6,7,8,9,p);
> }
>
> Let's say you compile this code with clang for x86_64 and look at the
> emitted LLVM IR and emitted assembly, e.g.
>
> clang -c -S file.c -O
> clang -c -S file.c -O -emit-llvm
>
> In the *.ll file you'll see this LLVM IR for 'foo':
>
> define i64 @foo(i64) local_unnamed_addr #0 {
>   %2 = tail call i64 @bar(i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64
> %0) #2
>   ret i64 %2
> }
>
> What is interesting here is that the function being called, "bar", has 7
> arguments. Under the rules of the C/C++ ABI for x86_64, not all of the
> parameters can be passed in registers -- some of them have to go in memory.
> However looking at the signature, there are no memory arguments, everything
> is being passed by value. Now let's look at the assembly:
>
> foo:
> pushq %rax
> *movq %rdi, (%rsp)*
> movl $1, %edi
> movl $2, %esi
> movl $3, %edx
> movl $4, %ecx
> movl $5, %r8d
> movl $6, %r9d
> callq bar
> popq %rcx
> retq
>
>
> As you can see, the back end knows about the rules of the C/C++ ABI for
> x86_64, and it has put the right things in registers and in memory so that
> the ABI is honored.
>
> So essentially what's happening is that the front end (clang) is playing a
> little game with the back end -- for the more complicated cases involving
> aggregates and things bigger than a single register, the front end makes
> the LLVM IR explicit with regard to what gets passed in memory via by
> value. For small parameters (things that will fit into a single register)
> it just leaves them by value and counts on having the back end sort it out.
> Doing things this way has advantages from a performance perspective.
>
> For your second question, yes, your approach seems reasonable --
> making CABIOracle into a base class and then adding virtual methods seems
> like the right way to go. I don't have any very specific advice on classes
> like CABIParamInfo and EightByteInfo -- I would say the former seems a bit
> more abstract and reusable whereas the notion of an "EightByte" seems
> pretty specific to the language in the x86_64 abi description. I would just
> take your best shot and see how things go.
>
> Hope this helps, let me know if you have other questions...
>
> Thanks, Than
>
>
>
>
> On Tue, Aug 20, 2019 at 8:42 AM eric fang  wrote:
>
>> Hi Than,
>>
>> I'm trying to implement the abi part for arm64, and got two questions for
>> you:
>>
>> 1, function CABIOracle::canPassDirectly in
>> gollvm/bridge/go-llvm-cabi-oracle.cpp, I'm not quite understand the first
>> if statement:
>>   if (regsInt + regsSSE == 1)
>>return true;
>>   Why not consider whether there are enough available registers here?
>>
>> 2, I plan to abstract the CABIOracle class into a base class, and each
>> architecture implements an inheritance class based on this class. But there
>> are some auxiliary classes and structures (like class CABIParamInfo and
>> EightByteInfo etc.) in this file that are also x86 specific. I haven't
>> figured out how to handle these yet, maybe some of them are also reusable.
>> Do you think this design is reasonable? Any suggestions? I should submit a
>> prototype patch, which would be more convenient to discuss, but I have not
>> implemented it yet. . .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/702e3129-b72a-411d-80fe-70e9daae207d%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 

Re: [go-nuts] How to build gollvm on arm platform

['Than McIntosh' via golang-nuts]

Hi Eric,

Thanks for the note. Your question about the code in
CABIOracle::canPassDirectly, e.g .

  if (regsInt + regsSSE == 1)
return true;

is a good one.  I'll see about adding some comments to the code there.

By way of explanation, the Gollvm code is modeled after the approach used
in clang, which does something similar in this case. For example, consider
the following small C program:

typedef unsigned long long u;
extern u bar(u p1, u p2, u p3, u p4, u p5, u p6, u p7, u p8, u p9, u p10);
u foo(u p) {
  return bar(1,2,3,4,5,6,7,8,9,p);
}

Let's say you compile this code with clang for x86_64 and look at the
emitted LLVM IR and emitted assembly, e.g.

clang -c -S file.c -O
clang -c -S file.c -O -emit-llvm

In the *.ll file you'll see this LLVM IR for 'foo':

define i64 @foo(i64) local_unnamed_addr #0 {
  %2 = tail call i64 @bar(i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 %0)
#2
  ret i64 %2
}

What is interesting here is that the function being called, "bar", has 7
arguments. Under the rules of the C/C++ ABI for x86_64, not all of the
parameters can be passed in registers -- some of them have to go in memory.
However looking at the signature, there are no memory arguments, everything
is being passed by value. Now let's look at the assembly:

foo:
pushq %rax
*movq %rdi, (%rsp)*
movl $1, %edi
movl $2, %esi
movl $3, %edx
movl $4, %ecx
movl $5, %r8d
movl $6, %r9d
callq bar
popq %rcx
retq


As you can see, the back end knows about the rules of the C/C++ ABI for
x86_64, and it has put the right things in registers and in memory so that
the ABI is honored.

So essentially what's happening is that the front end (clang) is playing a
little game with the back end -- for the more complicated cases involving
aggregates and things bigger than a single register, the front end makes
the LLVM IR explicit with regard to what gets passed in memory via by
value. For small parameters (things that will fit into a single register)
it just leaves them by value and counts on having the back end sort it out.
Doing things this way has advantages from a performance perspective.

For your second question, yes, your approach seems reasonable --
making CABIOracle into a base class and then adding virtual methods seems
like the right way to go. I don't have any very specific advice on classes
like CABIParamInfo and EightByteInfo -- I would say the former seems a bit
more abstract and reusable whereas the notion of an "EightByte" seems
pretty specific to the language in the x86_64 abi description. I would just
take your best shot and see how things go.

Hope this helps, let me know if you have other questions...

Thanks, Than




On Tue, Aug 20, 2019 at 8:42 AM eric fang  wrote:

> Hi Than,
>
> I'm trying to implement the abi part for arm64, and got two questions for
> you:
>
> 1, function CABIOracle::canPassDirectly in
> gollvm/bridge/go-llvm-cabi-oracle.cpp, I'm not quite understand the first
> if statement:
>   if (regsInt + regsSSE == 1)
>return true;
>   Why not consider whether there are enough available registers here?
>
> 2, I plan to abstract the CABIOracle class into a base class, and each
> architecture implements an inheritance class based on this class. But there
> are some auxiliary classes and structures (like class CABIParamInfo and
> EightByteInfo etc.) in this file that are also x86 specific. I haven't
> figured out how to handle these yet, maybe some of them are also reusable.
> Do you think this design is reasonable? Any suggestions? I should submit a
> prototype patch, which would be more convenient to discuss, but I have not
> implemented it yet. . .
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/702e3129-b72a-411d-80fe-70e9daae207d%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BUr55Eq3%3DDBHKAXJ3tmJbygAmO0th_kyPEer2BnViCysfbdJw%40mail.gmail.com.

Re: [go-nuts] Go string and UTF-8

[Jan Mercl]

On Tue, Aug 20, 2019 at 3:17 PM Pierre Durand  wrote:
>
> Well, in my case I don't want to convert the []byte to hexadecimal string, 
> because it uses 2x more memory.
> The code contains a huge map where the key is an MD5 hash.

md5 hash is an array type and can be used as a map key directly:
https://golang.org/pkg/crypto/md5/#Sum

Example: https://play.golang.org/p/qp-LFWh2Jln

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAA40n-XHXyf4qEeGEVCR2-86wUWzWTnA2y4myjJBZbaCfcAUdA%40mail.gmail.com.

[go-nuts] Running tests per package with Go module

[Dr Waryaa]

Hi,
I have a Go module which looks as follows:

go.mod
go.sum
 |
- pkg1
|
-pkg2

The module is called mydomain.com/mypkg

I'm trying to run the tests for pkg1 by calling

go test ./pkg1/...

My problem is that pkg1 imports mydomain.com/mypkg/pkg2 and that running 
the tests this way seems to mean that the go.mod is ignored and 
mydomain.com/mypkg/pkg2 is expected to be found in the GOPATH instead.

I suspect I could run

go test ...

but I would prefer running them one package at a time which was easy enough 
when using the GOPATH. Can I still do that with modules without having to 
install the package into $GOPATH/src/mydomain.com/mypkg ?

Cheers,
Chris

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/009382ff-10ea-4f2a-a66e-cd314c0711c5%40googlegroups.com.

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

[robert engels]

If you just print the final value of V in both cases you will see: On my iMac:

channels: 215606060
mutex: 651606060

Or the mutex has 3x greater throughput. How the Go routines are scheduled is OS 
dependent - the OSX code uses the same mechanism in both cases.

You have to remember that that the benchmark harness times the cost of 
operation when it runs, but it doesn’t always run the same number of 
invocations - it is based on the the scheduling of the Go routines, and the 
desired length of the test time.

goos: darwin
goarch: amd64
BenchmarkChanWrite/goroutines-4800-8 300   638 ns/op
BenchmarkChanWrite/goroutines-4880-8 200   636 ns/op
BenchmarkChanWrite/goroutines-4960-8 300   602 ns/op
BenchmarkChanWrite/goroutines-5040-8 200   632 ns/op
BenchmarkChanWrite/goroutines-5120-8 200   591 ns/op
BenchmarkChanWrite/goroutines-5200-8 300   634 ns/op
BenchmarkChanWrite/goroutines-5280-8 300   632 ns/op
BenchmarkChanWrite/goroutines-5360-8 300   581 ns/op
BenchmarkChanWrite/goroutines-5440-8 300   584 ns/op
BenchmarkChanWrite/goroutines-5520-8 300   582 ns/op
BenchmarkChanWrite/goroutines-5600-8 300   601 ns/op
BenchmarkChanWrite/goroutines-5680-8 300   609 ns/op
BenchmarkChanWrite/goroutines-5760-8 300   597 ns/op
BenchmarkChanWrite/goroutines-5840-8 300   674 ns/op
BenchmarkChanWrite/goroutines-5920-8 300   597 ns/op
BenchmarkChanWrite/goroutines-6000-8 300   601 ns/op
BenchmarkChanWrite/goroutines-6080-8 200   616 ns/op
BenchmarkChanWrite/goroutines-6160-8 300   603 ns/op
BenchmarkChanWrite/goroutines-6240-8 200   624 ns/op
BenchmarkChanWrite/goroutines-6320-8 300   603 ns/op
BenchmarkChanWrite/goroutines-6400-8 300   602 ns/op
BenchmarkChanWrite/goroutines-6480-8 200   616 ns/op
BenchmarkChanWrite/goroutines-6560-8 200   611 ns/op
BenchmarkChanWrite/goroutines-6640-8 300   603 ns/op
BenchmarkChanWrite/goroutines-6720-8 200   621 ns/op
BenchmarkChanWrite/goroutines-6800-8 300   636 ns/op
BenchmarkChanWrite/goroutines-6880-8 200   655 ns/op
BenchmarkChanWrite/goroutines-6960-8 300   605 ns/op
BenchmarkChanWrite/goroutines-7040-8 300   606 ns/op
BenchmarkChanWrite/goroutines-7120-8 200   639 ns/op
BenchmarkChanWrite/goroutines-7200-8 300   608 ns/op
BenchmarkChanWrite/goroutines-7280-8 300   617 ns/op
BenchmarkChanWrite/goroutines-7360-8 300   594 ns/op
BenchmarkChanWrite/goroutines-7440-8 300   596 ns/op
BenchmarkChanWrite/goroutines-7520-8 200   599 ns/op
BenchmarkChanWrite/goroutines-7600-8 200   610 ns/op
BenchmarkChanWrite/goroutines-7680-8 300   597 ns/op
BenchmarkChanWrite/goroutines-7760-8 200   608 ns/op
BenchmarkChanWrite/goroutines-7840-8 300   592 ns/op
BenchmarkChanWrite/goroutines-7920-8 300   611 ns/op
BenchmarkChanWrite/goroutines-8000-8 200   618 ns/op
BenchmarkChanWrite/goroutines-8080-8 300   612 ns/op
BenchmarkChanWrite/goroutines-8160-8 200   609 ns/op
BenchmarkChanWrite/goroutines-8240-8 300   594 ns/op
BenchmarkChanWrite/goroutines-8320-8 300   629 ns/op
BenchmarkChanWrite/goroutines-8400-8 200   624 ns/op
BenchmarkChanWrite/goroutines-8480-8 200   606 ns/op
BenchmarkChanWrite/goroutines-8560-8 200   635 ns/op
BenchmarkChanWrite/goroutines-8640-8 200   603 ns/op
BenchmarkChanWrite/goroutines-8720-8 300   625 ns/op
BenchmarkChanWrite/goroutines-8800-8 200   632 ns/op
BenchmarkChanWrite/goroutines-8880-8 200   602 ns/op
BenchmarkChanWrite/goroutines-8960-8 300   632 ns/op
BenchmarkChanWrite/goroutines-9040-8 200   602 ns/op
BenchmarkChanWrite/goroutines-9120-8 300   625 ns/op

Re: [go-nuts] Go string and UTF-8

[Pierre Durand]

Well, in my case I don't want to convert the []byte to hexadecimal string, 
because it uses 2x more memory.
The code contains a huge map where the key is an MD5 hash.

Please note that I'm not personally working on this.
I was reviewing the code written by a coworker, and I noticed that there 
was a string variable containing "invalid UTF-8 bytes".
It felt very strange to have a string containing invalid text.

So I have another question: since md5.Sum() is returning a [16]byte, is it 
better to use [16]byte as a map key ?
Or should I use a string containing invalid text as a map key ?

Le mardi 20 août 2019 14:01:38 UTC+2, Sam Whited a écrit :
>
>
>
> On August 20, 2019 11:50:54 AM UTC, Rob Pike  > wrote: 
> >Printf can print hexadecimal just fine. Never understood the point of 
> >encoding/hex. 
>
> I always thought that the C style format strings were unreadable and the 
> hex package methods were much clearer, personally. 
>
> —Sam 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4d600bee-5aae-454e-8419-cf6560af37bb%40googlegroups.com.

Re: [go-nuts] How to build gollvm on arm platform

[eric fang]

Hi Than,

I'm trying to implement the abi part for arm64, and got two questions for 
you:

1, function CABIOracle::canPassDirectly in 
gollvm/bridge/go-llvm-cabi-oracle.cpp, I'm not quite understand the first 
if statement:
  if (regsInt + regsSSE == 1)
   return true;
  Why not consider whether there are enough available registers here?

2, I plan to abstract the CABIOracle class into a base class, and each 
architecture implements an inheritance class based on this class. But there 
are some auxiliary classes and structures (like class CABIParamInfo and 
EightByteInfo etc.) in this file that are also x86 specific. I haven't 
figured out how to handle these yet, maybe some of them are also reusable. 
Do you think this design is reasonable? Any suggestions? I should submit a 
prototype patch, which would be more convenient to discuss, but I have not 
implemented it yet. . .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/702e3129-b72a-411d-80fe-70e9daae207d%40googlegroups.com.

Re: [go-nuts] Go string and UTF-8

[Sam Whited]

On August 20, 2019 11:50:54 AM UTC, Rob Pike  wrote:
>Printf can print hexadecimal just fine. Never understood the point of
>encoding/hex.

I always thought that the C style format strings were unreadable and the hex 
package methods were much clearer, personally.

—Sam

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4D22ED0E-1F1F-420B-8C26-3EF35F04EDCA%40samwhited.com.

Re: [go-nuts] Go string and UTF-8

[Rob Pike]

Printf can print hexadecimal just fine. Never understood the point of
encoding/hex.

Meanwhile, for questions about strings, see blog.golang.org/strings.

-rob


On Tue, Aug 20, 2019 at 9:00 PM Sam Whited  wrote:

> I personally wouldn't do this. If you're going to incur the overhead of a
> heap allocation, might as well incur a bit more and encode the hash, eg.
> using hex.EncodeToString [1], just so that you don't forget and try to
> print or decode the string as utf8 later.
>
> —Sam
>
> [1]: https://godoc.org/encoding/hex#EncodeToString
>
> On August 20, 2019 8:12:20 AM UTC, Pierre Durand 
> wrote:
> >I know that by convention Go string contain UTF-8 encoded text.
> >
> >Is it recommended/a good practice to store invalid bytes in a string ?
> >
> >The use case:
> >- compute a hash => get a []byte
> >- convert the []byte to string (this string is not UTF-8 valid)
> >- use the string as a map key
> >
> >In my case, the hash algorithm is md5.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/42989202-2854-479E-A536-13664BE41946%40samwhited.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOXNBZTKU%3DmSjj%2B9vujYFqphR0Zy%2BMtW0nHW2OayLchEgdvrEA%40mail.gmail.com.

Re: [go-nuts] Go string and UTF-8

[Sam Whited]

I personally wouldn't do this. If you're going to incur the overhead of a heap 
allocation, might as well incur a bit more and encode the hash, eg. using 
hex.EncodeToString [1], just so that you don't forget and try to print or 
decode the string as utf8 later.

—Sam

[1]: https://godoc.org/encoding/hex#EncodeToString

On August 20, 2019 8:12:20 AM UTC, Pierre Durand  wrote:
>I know that by convention Go string contain UTF-8 encoded text.
>
>Is it recommended/a good practice to store invalid bytes in a string ?
>
>The use case:
>- compute a hash => get a []byte
>- convert the []byte to string (this string is not UTF-8 valid)
>- use the string as a map key
>
>In my case, the hash algorithm is md5.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/42989202-2854-479E-A536-13664BE41946%40samwhited.com.

[go-nuts] Re: Go string and UTF-8

[Pierre Durand]

OK, thank you !

Le mardi 20 août 2019 10:34:55 UTC+2, djeg...@gmail.com a écrit :
>
> On Tue, Aug 20, 2019 at 10:12 AM Pierre Durand wrote:
> >
> > I know that by convention Go string contain UTF-8 encoded text.
>
> To my understanding this is not entirely true -- see 
> https://blog.golang.org/strings#TOC_2. -- It is simply a readonly slice 
> of bytes. However there is at least 2 places where UTF-8 encoding is used 
> for strings in the language spec: source code file is expected to be UTF-8 
> (thus string literals are partially influenced), and when using the `for 
> range` construct on a string. Otherwise there are various packages (e.g. 
> unicode/utf8) which expect UTF-8 encoded strings as arguments.
>
> > Is it recommended/a good practice to store invalid bytes in a string ?
>
> Thus the concept of _invalid bytes in a string_ doesn't really exist ;-).
>
> > The use case:
> > - compute a hash => get a []byte
> > - convert the []byte to string (this string is not UTF-8 valid)
> > - use the string as a map key
>
> I don't see any issues with this.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e8610962-732e-4744-9faf-c8581c444064%40googlegroups.com.

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

[changkun]

Hi Ian Davis, I read the issue before I post this thread. 
I think the old issue is quite different than here. Here the problem 
discusses sudden performance drop and unexpected regression, 
but the issue#5183 only did experiment on a very limited number of 
goroutines, and Dmitry's answer is fair in that case.

Best, Changkun

On Tuesday, August 20, 2019 at 10:55:13 AM UTC+2, Ian Davis wrote:
>
>
> On Tue, 20 Aug 2019, at 9:33 AM, changkun wrote:
>
> Hi Robert,
>
> Thanks for your explanation. But how could I "logged the number of 
> operations done per Go routine", which particular debug settings you 
> referring to?
> It is reasonable that sync.Mutex rely on runtime scheduler but channels do 
> not. However, it is unclear why a significant performance drop appears. Is 
> it possible to determine when the performance will appear?
>
>
> This comment by Dmitry Vyukov on a very old issue might help (I have no 
> idea if it is still valid after 6 years though) 
>
> If you are interested why chan does not have the same issue, runtime handles 
> chan's in a
> special way (active/passive spinning + thread parking instead of goroutine 
> parking),
> because it knowns that the critical section is bounded and small. For 
> sync.Mutex it does
> not have any knowledge as to critical section size.
>
>
> See https://github.com/golang/go/issues/5183
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e90669c4-42de-4f33-9327-41d35cf26077%40googlegroups.com.

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

[Ian Davis]

On Tue, 20 Aug 2019, at 9:33 AM, changkun wrote:
> Hi Robert,
> 
> Thanks for your explanation. But how could I "logged the number of operations 
> done per Go routine", which particular debug settings you referring to?
> It is reasonable that sync.Mutex rely on runtime scheduler but channels do 
> not. However, it is unclear why a significant performance drop appears. Is it 
> possible to determine when the performance will appear?

This comment by Dmitry Vyukov on a very old issue might help (I have no idea if 
it is still valid after 6 years though) 

If you are interested why chan does not have the same issue, runtime handles 
chan's in a
special way (active/passive spinning + thread parking instead of goroutine 
parking),
because it knowns that the critical section is bounded and small. For 
sync.Mutex it does
not have any knowledge as to critical section size.

See https://github.com/golang/go/issues/5183

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/55e8058b-cf7e-40c5-a81e-c3643a4ba507%40www.fastmail.com.

[go-nuts] Re: Go string and UTF-8

[djego . joss]

On Tue, Aug 20, 2019 at 10:12 AM Pierre Durand wrote:
>
> I know that by convention Go string contain UTF-8 encoded text.

To my understanding this is not entirely true -- see 
https://blog.golang.org/strings#TOC_2. -- It is simply a readonly slice of 
bytes. However there is at least 2 places where UTF-8 encoding is used for 
strings in the language spec: source code file is expected to be UTF-8 
(thus string literals are partially influenced), and when using the `for 
range` construct on a string. Otherwise there are various packages (e.g. 
unicode/utf8) which expect UTF-8 encoded strings as arguments.

> Is it recommended/a good practice to store invalid bytes in a string ?

Thus the concept of _invalid bytes in a string_ doesn't really exist ;-).

> The use case:
> - compute a hash => get a []byte
> - convert the []byte to string (this string is not UTF-8 valid)
> - use the string as a map key

I don't see any issues with this.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/34d7cce7-91d0-454f-ab1c-c373a984d66f%40googlegroups.com.

Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

[changkun]

Hi Robert,

Thanks for your explanation. But how could I "logged the number of 
operations done per Go routine", which particular debug settings you 
referring to?
It is reasonable that sync.Mutex rely on runtime scheduler but channels do 
not. However, it is unclear why a significant performance drop appears. Is 
it possible to determine when the performance will appear?

Best,
Changkun

On Monday, August 19, 2019 at 10:27:19 PM UTC+2, Robert Engels wrote:
>
> I think you'll find the reason that the Mutex uses the Go scheduler. The 
> chan is controlled by a 'mutex' which eventually defers to the OS futex - 
> and the OS futex is probably more efficient at scheduling in the face of 
> large contention - although you would think it should be the other way 
> around.
>
> I am guessing that if you logged the number of operations done per Go 
> routine, you will see that the Mutex version is very fair, and the 
> chan/futex version is unfair - meaning many are starved.
>
> -Original Message- 
> From: changkun 
> Sent: Aug 19, 2019 12:50 PM 
> To: golang-nuts 
> Subject: [go-nuts] sync.Mutex encounter large performance drop when 
> goroutine contention more than 3400 
>
> I am comparing the performance regarding sync.Mutex and Go channels. Here 
> is my benchmark: https://play.golang.org/p/zLjVtsSx9gd
>
> The performance comparison visualization is as follows:
>
> [image: sync.Mutex performance (1).png]
> What are the reasons that 
>
> 1. sync.Mutex encounter a large performance drop when the number of 
> goroutines goes higher than roughly 3400?
> 2. Go channels are pretty stable but slower than sync.Mutex before?
>
>
>
> Raw bench data by benchstat (go test -bench=. -count=5):
>
> MutexWrite/goroutines-2400-8  48.6ns ± 1%
> MutexWrite/goroutines-2480-8  49.1ns ± 0%
> MutexWrite/goroutines-2560-8  49.7ns ± 1%
> MutexWrite/goroutines-2640-8  50.5ns ± 3%
> MutexWrite/goroutines-2720-8  50.9ns ± 2%
> MutexWrite/goroutines-2800-8  51.8ns ± 3%
> MutexWrite/goroutines-2880-8  52.5ns ± 2%
> MutexWrite/goroutines-2960-8  54.1ns ± 4%
> MutexWrite/goroutines-3040-8  54.5ns ± 2%
> MutexWrite/goroutines-3120-8  56.1ns ± 3%
> MutexWrite/goroutines-3200-8  63.2ns ± 5%
> MutexWrite/goroutines-3280-8  77.5ns ± 6%
> MutexWrite/goroutines-3360-8   141ns ± 6%
> MutexWrite/goroutines-3440-8   239ns ± 8%
> MutexWrite/goroutines-3520-8   248ns ± 3%
> MutexWrite/goroutines-3600-8   254ns ± 2%
> MutexWrite/goroutines-3680-8   256ns ± 1%
> MutexWrite/goroutines-3760-8   261ns ± 2%
> MutexWrite/goroutines-3840-8   266ns ± 3%
> MutexWrite/goroutines-3920-8   276ns ± 3%
> MutexWrite/goroutines-4000-8   278ns ± 3%
> MutexWrite/goroutines-4080-8   286ns ± 5%
> MutexWrite/goroutines-4160-8   293ns ± 4%
> MutexWrite/goroutines-4240-8   295ns ± 2%
> MutexWrite/goroutines-4320-8   280ns ± 8%
> MutexWrite/goroutines-4400-8   294ns ± 9%
> MutexWrite/goroutines-4480-8   285ns ±10%
> MutexWrite/goroutines-4560-8   290ns ± 8%
> MutexWrite/goroutines-4640-8   271ns ± 3%
> MutexWrite/goroutines-4720-8   271ns ± 4%
>
> ChanWrite/goroutines-2400-8  158ns ± 3%
> ChanWrite/goroutines-2480-8  159ns ± 2%
> ChanWrite/goroutines-2560-8  161ns ± 2%
> ChanWrite/goroutines-2640-8  161ns ± 1%
> ChanWrite/goroutines-2720-8  163ns ± 1%
> ChanWrite/goroutines-2800-8  166ns ± 3%
> ChanWrite/goroutines-2880-8  168ns ± 1%
> ChanWrite/goroutines-2960-8  176ns ± 4%
> ChanWrite/goroutines-3040-8  176ns ± 2%
> ChanWrite/goroutines-3120-8  180ns ± 1%
> ChanWrite/goroutines-3200-8  180ns ± 1%
> ChanWrite/goroutines-3280-8  181ns ± 2%
> ChanWrite/goroutines-3360-8  183ns ± 2%
> ChanWrite/goroutines-3440-8  188ns ± 3%
> ChanWrite/goroutines-3520-8  190ns ± 2%
> ChanWrite/goroutines-3600-8  193ns ± 2%
> ChanWrite/goroutines-3680-8  196ns ± 3%
> ChanWrite/goroutines-3760-8  199ns ± 2%
> ChanWrite/goroutines-3840-8  206ns ± 2%
> ChanWrite/goroutines-3920-8  209ns ± 2%
> ChanWrite/goroutines-4000-8  206ns ± 2%
> ChanWrite/goroutines-4080-8  209ns ± 2%
> ChanWrite/goroutines-4160-8  208ns ± 2%
> ChanWrite/goroutines-4240-8  209ns ± 3%
> ChanWrite/goroutines-4320-8  213ns ± 2%
> ChanWrite/goroutines-4400-8  209ns ± 2%
> ChanWrite/goroutines-4480-8  211ns ± 1%
> ChanWrite/goroutines-4560-8  213ns ± 2%
> ChanWrite/goroutines-4640-8  215ns ± 1%
> ChanWrite/goroutines-4720-8  218ns ± 3%
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golan...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/3275fb21-dfbd-411d-be42-683386e7ebe2%40googlegroups.com
>  
> 
> .
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 

[go-nuts] Go string and UTF-8

[Pierre Durand]

I know that by convention Go string contain UTF-8 encoded text.

Is it recommended/a good practice to store invalid bytes in a string ?

The use case:
- compute a hash => get a []byte
- convert the []byte to string (this string is not UTF-8 valid)
- use the string as a map key

In my case, the hash algorithm is md5.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ec99fbdb-338b-4094-8b37-1d19b3851c72%40googlegroups.com.