[go-nuts] Re: [security] Go 1.7.4 and Go 1.6.4 are released

[Uli Kunitz]

I find it unfortunate that the fix for  #17276 
 "time: TestLoadFixed failure" 
has not been included in go1.7.4. So tests in all.bash (to buiild from 
source) will fail on a fully patched Ubuntu 14.04 LTS. I expect go tool 
dist test to fail on all Linux systems that have a recent tzdata update 
installed.There is go tool dist test -k to run all the test, but installing 
a binary with a broken test doesn't feel right. Since there is already an 
issue raised on the subject, I will not create a new one.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] fatal error: bad pointer in write barrier

[Ian Lance Taylor]

On Thu, Dec 1, 2016 at 4:47 PM, 'Florian Uekermann' via golang-nuts
 wrote:
>
> I just had a fairly strange not reproducible panic. I have no idea whether
> this could be a go bug or if the C library I am using is buggy. In case it
> looks like a go bug I should probably post this on the issue tracker. Does
> anyone have an idea whether this is useful or if I should just forget about
> it? This is what the bug report would look like:
>
> I build and ran a go program whose code I unfortunately can't share. The
> program uses a go library called def (see stacktrace) which makes extensive
> use of cgo. The line where it panics "def/instance.go:353" contains only an
> assignment. The assignment is of the form *X.Y, where Y is a pointer to
> space on the C heap that is allocated in the previous line.

> ### Stacktrace
> runtime: writebarrierptr *0x7fbca8000960 = 0x1
> fatal error: bad pointer in write barrier

You wrote `*X.Y`; I'm going to assume you meant to write `*X = Y`.

This error message is telling you that in the assignment `*X = Y` Y
has a pointer type, but has the value 1.  If you allocated Y from the
C heap, then it sounds like your C allocation failed with NULL and
somewhere along the way something added 1.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] fatal error: bad pointer in write barrier

['Florian Uekermann' via golang-nuts]

I just had a fairly strange not reproducible panic. I have no idea whether 
this could be a go bug or if the C library I am using is buggy. In case it 
looks like a go bug I should probably post this on the issue tracker. Does 
anyone have an idea whether this is useful or if I should just forget about 
it? This is what the bug report would look like:

I build and ran a go program whose code I unfortunately can't share. The 
program uses a go library called def (see stacktrace) which makes extensive 
use of cgo. The line where it panics "def/instance.go:353" contains only an 
assignment. The assignment is of the form *X.Y, where Y is a pointer to 
space on the C heap that is allocated in the previous line.

This happened once and never again. The program and go library are strictly 
single threaded. I don't know much about the C code around it though.

### What version of Go are you using (`go version`)?
$ go version
go version go1.7.3 linux/amd64

### What operating system and processor architecture are you using (`go 
env`)?
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/florian/go"
GORACE=""
GOROOT="/usr/lib/go-1.7"
GOTOOLDIR="/usr/lib/go-1.7/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 
-fdebug-prefix-map=/tmp/go-build640009790=/tmp/go-build 
-gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

### Stacktrace
runtime: writebarrierptr *0x7fbca8000960 = 0x1
fatal error: bad pointer in write barrier

runtime stack:
runtime.throw(0x55fdd48b2bd2, 0x1c)
/usr/lib/go-1.7/src/runtime/panic.go:566 +0x95
runtime.writebarrierptr.func1()
/usr/lib/go-1.7/src/runtime/mbarrier.go:151 +0xbd
runtime.systemstack(0xc42001c000)
/usr/lib/go-1.7/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
/usr/lib/go-1.7/src/runtime/proc.go:1079

goroutine 1 [running]:
runtime.systemstack_switch()
/usr/lib/go-1.7/src/runtime/asm_amd64.s:252 fp=0xc42003dd18 
sp=0xc42003dd10
runtime.writebarrierptr(0x7fbca8000960, 0x1)
/usr/lib/go-1.7/src/runtime/mbarrier.go:152 +0x7b fp=0xc42003dd50 
sp=0xc42003dd18
def.(*Instance).DrawLoop(0xc420062f70)
/home/florian/go/src/def/instance.go:353 +0xbea fp=0xc42003deb0 
sp=0xc42003dd50
main.main()
/home/florian/go/src/program/main.go:35 +0x27a fp=0xc42003df58 
sp=0xc42003deb0
runtime.main()
/usr/lib/go-1.7/src/runtime/proc.go:183 +0x1f4 fp=0xc42003dfb0 
sp=0xc42003df58
runtime.goexit()
/usr/lib/go-1.7/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42003dfb8 
sp=0xc42003dfb0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/lib/go-1.7/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall]:
runtime.CPUProfile(0x0, 0x0, 0x0)
/usr/lib/go-1.7/src/runtime/cpuprof.go:447 +0x2d
runtime/pprof.profileWriter(0x55fdd4b52a80, 0xc42002c078)
/usr/lib/go-1.7/src/runtime/pprof/pprof.go:676 +0x26
created by runtime/pprof.StartCPUProfile
/usr/lib/go-1.7/src/runtime/pprof/pprof.go:670 +0x122

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Can't understand the nil argument passing to function difference

[Ian Lance Taylor]

On Thu, Dec 1, 2016 at 1:46 PM, Sergey Parilin  wrote:
>
> Could anyone explain why it happens? Why nil checking is working for the
> first case and doesn't  work for rest of cases?

https://golang.org/doc/faq#nil_error

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] TestMissingGOPATHGetWarnsIfNotExists is failing for me behind a proxy on go1.8beta1

[Ian Lance Taylor]

On Thu, Dec 1, 2016 at 1:33 PM, Andrew O'Neill  wrote:
>
> I ran "./all.bash" on go1.8beta1 from my work machine and the test
> TestMissingGOPATHGetWarnsIfNotExists failed. The proxy settings are correct
> for git but not set in my environment. The output is below. Not sure why
> it's ignoring my git proxy settings.

> --- FAIL: TestMissingGOPATHGetWarnsIfNotExists (150.99s)
> go_test.go:260: running testgo [get -v github.com/golang/example/hello]
> go_test.go:279: standard error:
> go_test.go:280: github.com/golang/example (download)
> created
> GOPATH=/var/folders/fw/cyq4bf811qv73hk5gc8_12gjvjltwj/T/072077158/go; see
> 'go help gopath'
> # cd .; git clone https://github.com/golang/example
> /var/folders/fw/cyq4bf811qv73hk5gc8_12gjvjltwj/T/072077158/go/src/github.com/golang/example
> Cloning into
> '/var/folders/fw/cyq4bf811qv73hk5gc8_12gjvjltwj/T/072077158/go/src/github.com/golang/example'...
> fatal: unable to access 'https://github.com/golang/example/': Failed to
> connect to github.com port 443: Operation timed out

Thanks for the report.  Whatever the problem is with the proxy, that
test should not be run by all.bash.  Sent https://golang.org/cl/33802.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [security] Go 1.7.4 and Go 1.6.4 are released

[Chris Broadfoot]

Quick update: go1.7.4 was tagged with the wrong commit.

It has been updated from:
go1.7.4 0ad8bf4122de7396f771ed12f86934ea3177d6cf
to
go1.7.4 6b36535cf382bce845dd2d272276e7ba350b0c6b

If you built from the go1.7.4 tag at 0ad8b, the version will be incorrectly 
reported as "go1.7.3".

The binaries hosted at https://golang.org/dl are unaffected.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Downloading go1

[Edward Muller]

I could, but for other $reasons I'd rather not if I don't have to.

I will if it's lost to history though.

On Thu, Dec 1, 2016 at 2:10 PM James Bardin  wrote:

>
> Can you build it from source?
> The repo of course contains the entire history.
>
>
>
> On Thursday, December 1, 2016 at 4:57:30 PM UTC-5, freeformz wrote:
>
> For $reasons I need to download the original linux amd64 version of go1.
>
> Older go versions are stored under
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/
>
> I have retrieved go1.0.1 (
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.1.linux-amd64.tar.gz)
> and later successfully.
>
> But I can't seem to find the right key for go1. I've tried:
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.linux-amd64.tar.gz
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1-linux-amd64.tar.gz
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.linux-amd64.tar.gz
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0-linux-amd64.tar.gz
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.0.linux-amd64.tar.gz
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.0-linux-amd64.tar.gz
> 
>
> All respond with:
>  encoding='UTF-8'?>NoSuchKeyThe specified key
> does not exist.
>
> I've also tried the getting an index (
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go
> /),
> like you can with https://storage.googleapis.com/golang/ (where modern go
> releases are stored), but that just returns the same error.
>
> Can anyone point me to the right https://storage.googleapis.com url for
> the go1 release?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Downloading go1

[James Bardin]

Can you build it from source?
The repo of course contains the entire history. 



On Thursday, December 1, 2016 at 4:57:30 PM UTC-5, freeformz wrote:
>
> For $reasons I need to download the original linux amd64 version of go1.
>
> Older go versions are stored under 
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/
>
> I have retrieved go1.0.1 (
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.1.linux-amd64.tar.gz)
>  
> and later successfully. 
>
> But I can't seem to find the right key for go1. I've tried:
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.linux-amd64.tar.gz
>  
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1-linux-amd64.tar.gz
>  
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.linux-amd64.tar.gz
>  
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0-linux-amd64.tar.gz
>  
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.0.linux-amd64.tar.gz
>  
> 
>
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go/go1.0.0-linux-amd64.tar.gz
>  
> 
>
> All respond with: 
>  encoding='UTF-8'?>NoSuchKeyThe specified key 
> does not exist.
>
> I've also tried the getting an index (
> https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/go
>  
> /),
>  
> like you can with https://storage.googleapis.com/golang/ (where modern go 
> releases are stored), but that just returns the same error.
>
> Can anyone point me to the right https://storage.googleapis.com url for 
> the go1 release?
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] [security] Go 1.7.4 and Go 1.6.4 are released

[Chris Broadfoot]

Two security-related issues were recently reported, and to address these 
issues we have just released Go 1.6.4 and Go 1.7.4.

We recommend that all users update to one of these releases (if you're not 
sure which, choose Go 1.7.4).

The issues addressed by these releases are:

On Darwin, user's trust preferences for root certificates were not honored. 
If the user had a root certificate loaded in their Keychain that was 
explicitly not trusted, a Go program would still verify a connection using 
that root certificate.
This is addressed by https://golang.org/cl/33721, tracked in 
https://golang.org/issue/18141.
Thanks to Xy Ziemba for identifying and reporting this issue.

The net/http package's Request.ParseMultipartForm method starts writing to 
temporary files once the request body size surpasses the given "maxMemory" 
limit. It was possible for an attacker to generate a multipart request 
crafted such that the server ran out of file descriptors.
This is addressed by https://golang.org/cl/30410, tracked in 
https://golang.org/issue/17965.
Thanks to Simon Rawet for the report.

Downloads are available at https://golang.org/dl for all supported 
platforms.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Why does a "concurrent" Go GC phase appear to be stop-the-world?

[Will Sewell]

I just wrote up our investigation in this blog 
post: https://blog.pusher.com/golangs-real-time-gc-in-theory-and-practice/

It includes an animation of the GC algorithm that my colleague James Fisher 
made. I think this could be valuable as a standalone resource.

If any of you have any feedback, please let me know.

On Thursday, 27 October 2016 17:58:30 UTC+1, Dave Cheney wrote:
>
> Thanks for confirming that CL 23540 has reduced STW delays.
>
> On Friday, 28 October 2016 03:52:29 UTC+11, Will Sewell wrote:
>>
>> Are you referring to https://go-review.googlesource.com/#/c/23540/ or 
>> https://github.com/golang/go/issues/16528? If it's the former, then yes 
>> I have tried the benchmark with the HEAD version on master of the compiler, 
>> and it did bring the pause down to ~7.7ms. I was under the impression the 
>> latter issue has not been fixed yet, and is the reason the pause time was 
>> not even lower.
>>
>> On Wednesday, 26 October 2016 21:45:59 UTC+1, Dave Cheney wrote:
>>>
>>> Will, the changes has been in master for a few months now, are you able 
>>> to build master from source and see if this has addressed the issue? I'm 
>>> sure Rick and Austin would appreciate the feedback.
>>>
>>> On Thursday, 27 October 2016 01:46:47 UTC+11, Will Sewell wrote:

 Thanks for the information. I think it could well be caused by that. 
 Below is the screenshot of one of the periods of time where the mutator is 
 blocked.


 


 Note: pause times were as high as 15ms with the tracer enabled.

 Similar sizes occur every ~100ms.

 Let's hope this gets resolved in Go1.8 :)

 On Monday, 24 October 2016 17:06:59 UTC+1, rhys.h...@gmail.com wrote:
>
> Yes, this sounds like https://golang.org/issue/16528. During the 
> concurrent mark phase (the "27 [ms]" of "0.008+27+0.072 ms clock"), both 
> your code and the garbage collector are running. The program is allowed 
> to 
> use four OS threads ("4 P"), which might be executing your code in your 
> goroutines, or might be running GC code in dedicated GC goroutines.
>
> There's plenty of work for the GC to do, so when a GC helper goroutine 
> is allowed to have some processing time it'll keep running until it has 
> used up all of its allowed time—ten milliseconds. If all four threads end 
> up running GC goroutines at the same time, your goroutine will need to 
> wait 
> until one of them has run for about 10ms before it can be scheduled 
> again. 
> This can lead to individual goroutines being paused for up to 10ms.
>
> You might be able to see this with the execution tracer, but it's not 
> an easy tool to use. See package "runtime/trace" and the command "go tool 
> trace" for some hints. Issue 16528 includes some screenshots of the tool.
>
> On Thursday, October 20, 2016 at 6:36:27 AM UTC-7, Will Sewell wrote:
>>
>> Interesting, that does sound like it could be the cause.
>>
>> I just tried running the same benchmark on master 
>> (692df217ca21b6df8e4dc65538fcc90733e8900e), and I get the following 
>> results:
>>
>> gc 1 @0.004s 3%: 0.009+0.41+0.049 ms clock, 0.036+0.11/0.36/0.12+0.19 
>> ms cpu, 4->4->3 MB, 5 MB goal, 4 P
>> gc 2 @0.008s 4%: 0.008+0.80+0.035 ms clock, 
>> 0.034+0.097/0.67/0.16+0.14 ms cpu, 7->7->7 MB, 8 MB goal, 4 P
>> gc 3 @0.016s 3%: 0.010+0.91+0.044 ms clock, 0.041+0/0.31/0.79+0.17 ms 
>> cpu, 13->15->14 MB, 15 MB goal, 4 P
>> gc 4 @0.032s 3%: 0.009+2.3+0.10 ms clock, 0.037+0.60/2.0/0.12+0.40 ms 
>> cpu, 27->28->27 MB, 29 MB goal, 4 P
>> gc 5 @0.070s 3%: 0.010+7.6+0.068 ms clock, 0.043+0.79/5.4/8.5+0.27 ms 
>> cpu, 51->53->51 MB, 54 MB goal, 4 P
>> gc 6 @0.149s 3%: 0.020+8.2+0.12 ms clock, 0.081+0.56/7.2/9.7+0.48 ms 
>> cpu, 98->102->99 MB, 103 MB goal, 4 P
>> gc 7 @0.282s 4%: 0.028+21+0.082 ms clock, 0.11+10/20/1.9+0.32 ms cpu, 
>> 190->195->190 MB, 198 MB goal, 4 P
>> gc 8 @0.568s 3%: 0.024+24+0.080 ms clock, 0.098+0/23/41+0.32 ms cpu, 
>> 364->376->214 MB, 381 MB goal, 4 P
>> gc 9 @0.816s 3%: 0.008+27+0.072 ms clock, 0.035+0/25/34+0.29 ms cpu, 
>> 412->420->213 MB, 428 MB goal, 4 P
>> gc 10 @1.064s 3%: 0.009+31+0.10 ms clock, 0.039+6.1/26/33+0.41 ms 
>> cpu, 415->427->216 MB, 427 MB goal, 4 P
>>
>> My manually calculated worst time for a call to mkMessage is 7.73812ms, 
>> which is much better than before. It's significantly faster than the 
>> worst 
>> wall clock time for the concurrent mark/scan phase, but it's also much 
>> slower than the worst STW phase. Do you know why this might be?
>>
>> Best,
>> Will
>>
>> On Wednesday, 19 October 2016 17:29:23 UTC+1, rhys.h...@gmail.com 

[go-nuts] Re: Weird behavior C.GoString

[Felipe Santos]

The string is: mismatched parenthesis

Em quinta-feira, 1 de dezembro de 2016 16:32:42 UTC-2, Felipe Santos 
escreveu:
>
> Sure,
> If I compare the two strings with a Go compare function it is different 
> because of the last char
>
> Em quinta-feira, 1 de dezembro de 2016 16:25:45 UTC-2, Didier Spezia 
> escreveu:
>>
>> Sorry for the silly question, but since you do not have any \n or 
>> separator after the %X,
>> are you sure the extra characters do not come from the next fmt.Printf in 
>> your code?
>>
>> On Thursday, December 1, 2016 at 5:25:41 PM UTC+1, Felipe Santos wrote:
>>>
>>> Hi,
>>>
>>> I am trying to link C++ shared lib to my Go program.
>>> When I convert a C string (* char) to Go string using C.GoString it adds 
>>> an extra char, in a not deterministic way.
>>> I print the bytes of my string inside C++ shared lib and it's correct, 
>>> with \0 on the end.
>>>
>>> //Shared lib code
>>>
>>> for(int j = 0; j < 23; j++) {
>>>   std::cout << std::hex <<   (int)extractor_result->error_message[j] << " " 
>>> ;
>>> }
>>>
>>> //Go Code
>>>
>>> fmt.Printf("%X", C.GoString(extractorResult.error_message))
>>>
>>>
>>>
>>> Shared lib  always prints:
>>>
>>> 6d 69 73 6d 61 74 63 68 65 64 20 70 61 72 65 6e 74 68 65 73 69 73 0
>>>
>>> Golang:
>>> printf prints:
>>>
>>> 6D69736D61746368656420706172656E746865736973*08*
>>> or
>>>
>>> 6D69736D61746368656420706172656E746865736973*17*
>>> or
>>> 6D69736D61746368656420706172656E746865736973*07*
>>> or
>>> 6D69736D61746368656420706172656E746865736973*03*
>>> and so on.
>>>
>>> Any tips about this issue?
>>>
>>> Thanks
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Weird behavior C.GoString

[Felipe Santos]

Sure,
If I compare the two strings with a Go compare function it is different 
because of the last char

Em quinta-feira, 1 de dezembro de 2016 16:25:45 UTC-2, Didier Spezia 
escreveu:
>
> Sorry for the silly question, but since you do not have any \n or 
> separator after the %X,
> are you sure the extra characters do not come from the next fmt.Printf in 
> your code?
>
> On Thursday, December 1, 2016 at 5:25:41 PM UTC+1, Felipe Santos wrote:
>>
>> Hi,
>>
>> I am trying to link C++ shared lib to my Go program.
>> When I convert a C string (* char) to Go string using C.GoString it adds 
>> an extra char, in a not deterministic way.
>> I print the bytes of my string inside C++ shared lib and it's correct, 
>> with \0 on the end.
>>
>> //Shared lib code
>>
>> for(int j = 0; j < 23; j++) {
>>   std::cout << std::hex <<   (int)extractor_result->error_message[j] << " " ;
>> }
>>
>> //Go Code
>>
>> fmt.Printf("%X", C.GoString(extractorResult.error_message))
>>
>>
>>
>> Shared lib  always prints:
>>
>> 6d 69 73 6d 61 74 63 68 65 64 20 70 61 72 65 6e 74 68 65 73 69 73 0
>>
>> Golang:
>> printf prints:
>>
>> 6D69736D61746368656420706172656E746865736973*08*
>> or
>>
>> 6D69736D61746368656420706172656E746865736973*17*
>> or
>> 6D69736D61746368656420706172656E746865736973*07*
>> or
>> 6D69736D61746368656420706172656E746865736973*03*
>> and so on.
>>
>> Any tips about this issue?
>>
>> Thanks
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: net/http and errors.

[mb . dhananjay]

> if err, ok := err.(*url.Error); ok {
> if err, ok := err.Err.(*net.OpError); ok {
> if err, ok := err.Err.(*net.DNSError); ok {
> fmt.Println("DNS ERROR:", err)
> }
> }
> }
>

This works as expected. :-)

On Monday, November 28, 2016 at 9:32:55 PM UTC+1, Dave Cheney wrote:
>
> Indeed. Does your program differentiate between a URL not resolving, the 
> webserver at the other end being down or refusing to connect, or it 
> returnining a non temporary fatal error, 4xx? 


Yes, it tries to be verbose about failure reasons. DNS / Server not 
listening / HTTP Status codes if present.. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Weird behavior C.GoString

[Didier Spezia]

Sorry for the silly question, but since you do not have any \n or separator 
after the %X,
are you sure the extra characters do not come from the next fmt.Printf in 
your code?

On Thursday, December 1, 2016 at 5:25:41 PM UTC+1, Felipe Santos wrote:
>
> Hi,
>
> I am trying to link C++ shared lib to my Go program.
> When I convert a C string (* char) to Go string using C.GoString it adds 
> an extra char, in a not deterministic way.
> I print the bytes of my string inside C++ shared lib and it's correct, 
> with \0 on the end.
>
> //Shared lib code
>
> for(int j = 0; j < 23; j++) {
>   std::cout << std::hex <<   (int)extractor_result->error_message[j] << " " ;
> }
>
> //Go Code
>
> fmt.Printf("%X", C.GoString(extractorResult.error_message))
>
>
>
> Shared lib  always prints:
>
> 6d 69 73 6d 61 74 63 68 65 64 20 70 61 72 65 6e 74 68 65 73 69 73 0
>
> Golang:
> printf prints:
>
> 6D69736D61746368656420706172656E746865736973*08*
> or
>
> 6D69736D61746368656420706172656E746865736973*17*
> or
> 6D69736D61746368656420706172656E746865736973*07*
> or
> 6D69736D61746368656420706172656E746865736973*03*
> and so on.
>
> Any tips about this issue?
>
> Thanks
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Does there exist a curated list of all Go Gopher images?

[Jack]

Is there a list/page/album/etc somewhere of every reusable Go Gopher image 
and it's associated license/author?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: New Proposal: Add support for pprof profiler labels

[Michael Matloob]

After a lot of discussion, I've updated the proposal issue with a new API
design. Please comment if you have feedback.

Issue: golang.org/issue/17280

Michael

On Tue, Oct 18, 2016 at 11:38 AM Michael Matloob  wrote:

Hello Gophers!


I've proposed adding a mechanism to the runtime profile support to allow
setting pprof labels. Profile labels are already supported by go tool
pprof, but there's no way of setting them yet.

Doc is here:
https://github.com/golang/proposal/blob/master/design/17280-profile-labels.md
Provide your comments here: https://github.com/golang/go/issues/17280

Thanks!
Michael

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: net.Conn.Read() performance

[James Bardin]

Conn.Read is a much higher level construct the using the syscalls directly. 
 The largest part of the overhead is coordinating with the network poller. 
There's also a slight overhead of using the net.Conn interface, so 
asserting the value to a *net.TCPConn may save you a little (though I'm not 
certain how much of a difference that makes in recent releases).

Using the syscalls, you're essentially dedicating a single OS thread to 
handle that socket's IO. 


On Wednesday, November 30, 2016 at 2:13:10 PM UTC-5, anto...@gmail.com 
wrote:
>
> Any thoughts, anyone?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go test -bench -benchmem, allocs/op mystery

[Erwin]

> The build flag -gcflags=-m will cause the compiler to report which 
> allocations escape to the heap.
>
> Give that a try and see what it tells you.

It doesn't report any escape info for that function. I did figure it
out by outcommenting parts of the function, it turns out that

scratch := f32scratch.Get(r2)
trow := scratch.buf[:r2]

are allocated on the heap, for some reason. Both are []float32 slices.
Oh, now i write this email i understand why the scratch buffer
escapes: at the end of the function it is put back on the sync.Pool
via a function call...
Thanks anyway!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Weird behavior C.GoString

[felipegs]

Hi,

I am trying to link C++ shared lib to my Go program.
When I convert a C string (* char) to Go string using C.GoString it adds an 
extra char, in a not deterministic way.
I print the bytes of my string inside C++ shared lib and it's correct, with 
\0 on the end.

//Shared lib code

for(int j = 0; j < 23; j++) {
  std::cout << std::hex <<   (int)extractor_result->error_message[j] << " " ;
}

//Go Code

fmt.Printf("%X", C.GoString(extractorResult.error_message))



Shared lib  always prints:

6d 69 73 6d 61 74 63 68 65 64 20 70 61 72 65 6e 74 68 65 73 69 73 0

Golang:
printf prints:

6D69736D61746368656420706172656E746865736973*08*
or

6D69736D61746368656420706172656E746865736973*17*
or
6D69736D61746368656420706172656E746865736973*07*
or
6D69736D61746368656420706172656E746865736973*03*
and so on.

Any tips about this issue?

Thanks


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: locks and "happens before" withing a goroutine

[Jesper Louis Andersen]

On Thu, Dec 1, 2016 at 10:38 AM Daniel Fanjul <
daniel.fanjul.alcu...@gmail.com> wrote:

>
> > Inside a single goroutine the code proceeds in statement order.
> The code generated by the compiler, right? Of course, it is typically the
> processor who reorders. But what is the limit of the reorder?
>

The memory model describes a "virtual" CPU which happens to execute
operations in-order for each go-routine. Further it sets up a (causal)
happens-before relationship when synchronizing several goroutines and their
in-order execution. In short, the memory model is a pure logical
construction, not bound to real hardware.

In Computer Science, people tend to belong to either the "theory" side of
things, where you describe algorithms and phenomenon purely detached from
the underlying hardware. Or you belong to the "practical" side of things,
where the hardware is central and you build up the logic from what is
possible with the hardware.

The memory model is where these two worlds meet.

The task for a compiler/runtime writer is to take the virtual model, and
map it onto real hardware. We are allowed a certain amount of leverage
here, as long as we make sure we obey the memory model in the end.
Specifically:

* We can allow the CPU to reorder memory writes, as long as we make sure
there is a memory-barrier instruction inserted in the right place by the
compiler or runtime

* We can allow the compiler to reorder instructions as well, as long as we
have a proof that the reordering preserves the semantics of the program.
Again, we would have to insert a memory barrier in places.

But note if we compile Go to hardware which has in-order execution, one
core, and where there is no cache and no coherency problems, we don't have
to insert a memory-barrier. Hence, the memory model is best written to not
contain a word such as "memory barrier". They are artifacts of the
"practical" hardware, not of the "theoretic" memory-model.

The happens-before relation defines a causal relation between different
goroutines. This is powerful since this is where the leverage is hidden: we
can have a goroutine run as a "soup" of reorderings in limbo until
communication happens. At the point of rendezvous the happens-before
relationship "locks down" the soup and makes sure it is in the right order.
This is often ensured via a memory barrier instruction (or one which that
side-effect). Since we can reorder freely until lock-down occur, we can
optimize code by simultaneous execution of independent statements
(out-of-order execution via Tomasulo's algorithm).

The key is that this actually models the real world. You may get a summary
of a meeting with the findings, but you don't care too much about what was
discussed in the meeting in order to reach those findings. In short, the
discussions are allowed "reordering" leverage, as long as the findings are
locked down in the summary when you hear about them. I.e., there is a
memory barrier happening at the end of the meeting.

Another real world example, of a causal violation this time, is when you
have an image service such as Instagram, Google Images, Snapchat, ...,
where a person first blocks a recipient and then posts an image to
everyone. Because the block happens-before the post, you don't want that
image to end up in the inbox of the blocked person. But if you violate the
causal dependency, then that is exactly what might happen: the image gets
posted first, then the block is instated. Many distributed systems violate
these orderings unless you are careful.

So what is the limit of the memory reorder? The limit is that the causal
(happen-before) consistency must not be violated. You are allowed, as in
the meeting example, to have a *local* violation, as long as external
observers are not able to see that local violation. This allows you to
optimize programs for efficiency, which is a good thing[0]. But that is all
you are allowed.

[0] There is an interesting observation here: local violations
as-long-as-observers-cannot-see-it is the central aspect of APIs and
interfaces. Which has similarity to assuming the truth of a theorem or
lemma in mathematics. Give a module a (purely) functional interface, but in
the actual implementation, use every imperative programming speed trick you
know of, violating FP in the process. The program is still "functional",
though it allows you to violate that property locally.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] synchronise read from two channels

[omarshariffdontlikeit]

Yup I thin I was frying my brain on this - it also led to questions such as 
"if I read a worker from the channel and there ISNT a URL waiting how do I 
write back into the channel making it available - what if more than one go 
routine is reading from the channel... etc"

I knew I was over complicating it I just couldnt see the forest for the 
trees! I assumed I wanted channels for the available and busy workers when 
really I wanted stacks.

thanks for the in-depth discussion - in future I will sleep on it next time 
and come at it afresh :)

(ironically I'd done exactly this before with stacks in an old project 
which is what gave me the lightbulb moment this morning)

Thanks everyone!

On Thursday, December 1, 2016 at 3:21:46 PM UTC, Jesper Louis Andersen 
wrote:
>
> On Wed, Nov 30, 2016 at 6:07 PM  
> wrote:
>
>> Hi, I'm having a bit of a slow day... I'm trying to synchronise two reads 
>> from two channels and can't get my fuzzy head round the problem.
>>
>>
> Go doesn't have support for an (atomic) read from multiple channels at 
> once. There are other systems which can do this (JoCaml, join-calculus, 
> Concurrent ML, Haskell's STM, Reagents[0] in Scala, ...)
>
> However, as you found out, one can often rearrange the code base as to 
> avoid the problem. There is also the problem of more advanced 
> select-constructs not being free in the performance sense: atomic sync on 
> multiple channels tend to be rather expensive.
>
> One reason it isn't that simple to implement is to consider the following 
> (hypothetical) snippet:
>
> select {
> case w := <-available && urlChan <- value:
> ..BODY..
> }
> }
>
> in which the atomic operation must simultaneously receive a message on the 
> 'available' channel and send a message on the 'urlChan' channel. And 
> further, since you introduced && to mean logical-and-between-channels, you 
> might as well introduce || to mean logical-or-between-channels. Once there, 
> you may want to introduce a new type
>
> event t
>
> for some type t alongside 'chan t'. Hence the expression (<-available) 
> becomes an expression of type 'event *worker' and now they can be composed 
> via && and ||. This means we can get rid of select since it is just an 
> expression which successively applies || to a slice of values. If we then 
> introduce an new statement, sync, which turns 'event t' into a 't', we are 
> done. And we have almost implemented the basis of Concurrent ML in the 
> process.
>
> In short, you could be opening a regular can of worms if you start 
> allowing for more complicated select-expressions. Hence the school of 
> keeping them relatively simple.
>
> [0] https://people.mpi-sws.org/~turon/reagents.pdf 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] synchronise read from two channels

[Jesper Louis Andersen]

On Wed, Nov 30, 2016 at 6:07 PM  wrote:

> Hi, I'm having a bit of a slow day... I'm trying to synchronise two reads
> from two channels and can't get my fuzzy head round the problem.
>
>
Go doesn't have support for an (atomic) read from multiple channels at
once. There are other systems which can do this (JoCaml, join-calculus,
Concurrent ML, Haskell's STM, Reagents[0] in Scala, ...)

However, as you found out, one can often rearrange the code base as to
avoid the problem. There is also the problem of more advanced
select-constructs not being free in the performance sense: atomic sync on
multiple channels tend to be rather expensive.

One reason it isn't that simple to implement is to consider the following
(hypothetical) snippet:

select {
case w := <-available && urlChan <- value:
..BODY..
}
}

in which the atomic operation must simultaneously receive a message on the
'available' channel and send a message on the 'urlChan' channel. And
further, since you introduced && to mean logical-and-between-channels, you
might as well introduce || to mean logical-or-between-channels. Once there,
you may want to introduce a new type

event t

for some type t alongside 'chan t'. Hence the expression (<-available)
becomes an expression of type 'event *worker' and now they can be composed
via && and ||. This means we can get rid of select since it is just an
expression which successively applies || to a slice of values. If we then
introduce an new statement, sync, which turns 'event t' into a 't', we are
done. And we have almost implemented the basis of Concurrent ML in the
process.

In short, you could be opening a regular can of worms if you start allowing
for more complicated select-expressions. Hence the school of keeping them
relatively simple.

[0] https://people.mpi-sws.org/~turon/reagents.pdf

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] a gameserver framework with golang

[sydnash1]

hello everyone:
   i'm a new for golang，and recently i write a gameserver framework 
which is like skynet. the git repository is   
http://github.com/sydnash/lotou
   It  contains a slave/master mode RPC and communication， and each 
service can be run in one goroutine.

  It maybe poor, but I wish some one can read the code and give some 
advice of the framework, and help me make it great. thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go @ Work

[Kiswono Prayogo]

Hi, if I may ask, what language do your company previously use before 
switch to Golang?

On Saturday, 31 March 2012 00:27:36 UTC+7, hcatlin wrote:
>
> Just thought I'd stop in and post something saying "hi" to the group. 
> I work at 
> Moovweb where we power some of the biggest e-commerce sites' mobile 
> experience 
> on our platform. Our platform has super strong demands on it and our 
> business is 
> growing at a nearly exponential rate. 
>
> This past summer, I made the decision to switch all of our internal 
> software to Go 
> after being persuaded by our Senior Architect, Zhigang Chen. Of 
> course, knowing that Go 
> was beta and that the future of the company was being bet on Go was 
> (and is) amazingly 
> stressful. 
>
> The library ecosystem was a little weak for our needs when we first 
> showed up so 
> we've since released a Regexp library called Rubex that makes regular 
> expressions 
> 2-3x faster, we've released Gokogiri which is a libxml wrapper library 
> for XML/HTML 
> parsing, and GVM the go version manager so that we can work with code 
> bases 
> on various versions of Go. 
>
> A couple observations: 
> * New users/employees have learned Go very quickly. +20! 
> * Found out about breakage in 32-bit while in production. -10! 
> * Had to write our own libraries for some basic stuff. -5! 
> * Performs very well in production (only on 64-bit). +10! 
>
> Man, so much to say... this bullet list might go on and on. 
>
> I mostly just wanted to say that to us Go is not a hobbyist language. 
> Its a complex 
> build environment for us with dependency management and 5 full time 
> developers 
> writing code daily in it. 
>
> We didn't build "some small part" of our platform in Go... its the 
> heart of our tech. 
> Our SDK is a compiled binary that we are packaging and sending out to 
> our 
> clients and integration partners. Our production system is handling 
> millions of pages 
> a day and millions of dollars of ecommerce a month. 
>
> We've bet on Go big time... and will continue to bet on it. We will 
> continue to release 
> new code and we will continue to try and mature the community and tool 
> set so that 
> we can deal with real life issues. 
>
> I'd love to hear if other companies are doing a similar thing. 
>
> http://site.moovweb.com // public site 
> http://github.com/moovweb // public software repo 
>
> -hampton (@hcatlin) 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] December Blog Post Series

[Brian Ketelsen]

The December Blog Post series has begun.  We won't post daily links here, but 
you can catch them all at https://blog.gopheracademy.com

The introductory post is here:  
https://blog.gopheracademy.com/advent-2016/introduction/

Special thanks to Damian Gryski for coordinating and herding cats this year to 
make this happen.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go is two times slower than python code?

[Nate Finch]

Generally, the answer is, don't use regular expressions.   This is good 
advice in any language and for almost any problem.  In this specific case, 
if you care about speed, write some more specific code that does what you 
need instead of falling back on regular expressions.

On Wednesday, November 30, 2016 at 4:59:02 AM UTC-5, Thomas Modeneis wrote:
>
> +6 years later, It seems that this issue still a problem.
> There is a open ticket on github for it: 
> https://github.com/golang/go/issues/16791
>
> I've been facing this issue for a while, and on my experience the best 
> workaround is to serialize the CSV to the disk after reading it.
>
> Cheers.
>
> On Tuesday, March 9, 2010 at 3:52:16 PM UTC+1, Alex Dong wrote:
>>
>> I'm writing a simple 'word picker' using golang and python. It
>> basically reads a tweeter message from a csv file, tokenize it and put
>> the word->id into a map called lexicon.  It turned out that same logic
>> took 22 seconds for python 2.6 to finish whereas 57 seconds for
>> golang!
>>
>> Wondering are there anything I've done wrong?
>>
>> Here is the python code:
>> $ cat loader.py
>> lexicon = {}
>> pnct_ptn = re.compile(r'([\.,\\/\'"\?!=_\)\(\]\[\{\}:;]+|http://[^ ]
>> +)')
>> def tokenize(s):
>> s = pnct_ptn.sub(' ', s)
>> return [t for t in s.split() if len(t)>=3]
>>
>> for line in open("result.csv").readlines():
>> parts = line.split(',', 3)
>> if len(parts) != 4: continue
>> msg = parts[3]
>> s  = msg.decode('utf8','ignore').lower()
>> for word in tokenize(s):
>> if not lexicon.has_key(word):
>> unique_words += 1
>> lexicon[word] = unique_words
>>
>> Here is the go code:
>> $ cat loader.go
>> package main
>>
>> import (
>> "bufio"
>> "os"
>> "regexp"
>> "strings"
>> )
>>
>> var (
>> pr, _ = regexp.Compile(`(http://[^ ]+|['".\\,=()*:;?!/]|-)`)//
>> pattern for removal
>> )
>>
>> func tokenize(s string) []string {
>> ms := pr.ReplaceAllString(strings.ToLower(s), " ")
>> return strings.Split(ms, " ", 0)
>> }
>>
>> func main() {
>> lex := make(map[string] int)// lexicon
>> dic := make(map[int] string)// lookup
>> tw  := 0// total
>> words
>> ps  := false// present
>>
>> r, _ := os.Open("result.csv", os.O_RDONLY, 0444)
>> defer r.Close()
>> in := bufio.NewReader(r)
>>
>> for i := 0; i >= 0; i++ {
>> line, err := in.ReadString('\n')
>> if err != nil {
>> break
>> }
>>
>> parts:= strings.Split(line, ",", 4)
>> if len(parts) != 4 {
>> continue
>> }
>>
>> ts := tokenize(parts[3])
>> for d := 0; d < len(ts); d++ {
>> w := ts[d]
>> if len(w) < 3 {
>> continue
>> }
>> _, ps = lex[w]
>> if ps == false {
>> lex[w] = tw
>> dic[tw] = w
>> tw ++
>> }
>> }
>> }
>> }
>>
>>
>> Cheers,
>> Alex
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: synchronise read from two channels

[omarshariffdontlikeit]

Just as a follow up for anyone else - in the end I used two simple stacks 
for available and active lists of workers and simply popped a worker of the 
available stack when a url came in, assigned it to the worker and then 
pushed it to the active stack. Job done!

On Thursday, December 1, 2016 at 9:07:42 AM UTC, omarsharif...@gmail.com 
wrote:
>
> Yeah I kinda figured I was coming to this wrong way - my tired brain had 
> an unterminated loop. Thanks for the pointers! (all puns intended)
>
> On Wednesday, November 30, 2016 at 5:08:06 PM UTC, omarsharif...@gmail.com 
> wrote:
>>
>> Hi, I'm having a bit of a slow day... I'm trying to synchronise two reads 
>> from two channels and can't get my fuzzy head round the problem.
>>
>> I have a channel containing urls (which will be feed periodically by a 
>> named pipe by another program) and I have a cannel which consists of a pool 
>> of available workers. I'd like to synchronise these and put a worker into a 
>> third channel for active workers if there is a url in the channel and if 
>> there is an available worker.
>>
>> Ideally I'd be able to do something like:
>>
>> go func(urlChan <-chan *url.URL, availableWorkerChan <-chan *worker, 
>> activeWorkerChan chan<- *worker){
>> for {
>> select{
>> case w := <-availableWorkerChan && w.Url = <-urlChan:
>> activeWorkersChan <- w
>> }
>> }
>> }(a, b, c)
>>
>> But obviously thats not right!
>>
>> Can anyone offer my tired brain a clue?
>>
>> Cheers!
>> Ben
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Help me beat Erlang, SkynetBenchmark challenge

[Michael Jones]

A variation, slightly faster:

https://play.golang.org/p/OzzJWRu6KL

 

One problem with “do nothing” benchmarks is that they test limits that are not 
real-life limits. I get about 3 million channel sends per second on my laptop 
and your benchmark is hitting that or else close to it. If you were doing 
something in the leaf…animating a character, simulating a missile, etc., then 
you would not measure the subtle differences in the implementations. As it is, 
the differences in program load time and other run-to-run variations are also 
statistically big here. The Go test framework is helpful in this regard, but 
what is really helpful is if the “model” of the test is legitimate. In this 
case, have your actors do whatever the real actors would do so that you are 
measuring “parallel actors doing.”

 

From: Michael Jones 
Date: Thursday, December 1, 2016 at 3:24 AM
To: Roger Alsing , golang-nuts 

Subject: Re: [go-nuts] Help me beat Erlang, SkynetBenchmark challenge

 

Here is an idea for you. 35% faster on my macbook pro:

https://play.golang.org/p/3Mb5pR0V0J

 

Michael

 

 

From:  on behalf of Roger Alsing 

Date: Thursday, December 1, 2016 at 12:52 AM
To: golang-nuts 
Subject: [go-nuts] Help me beat Erlang, SkynetBenchmark challenge

 

I am currently trying to optimize the in process part of my Actor Model 
framework for Go - GAM.

In order to compare it to Erlang and Akka, I am using the "Skynet Benchmark" 
https://github.com/atemerev/skynet#results-i7-4771-ubuntu-1510

 

I have ported this test to my lib: 
https://github.com/AsynkronIT/gam/blob/dev/examples/spawnbenchmark/main.go

It works and the performance is roughly the same as the Scala Akka performace.

 

I have noticed that if I turn of GOGC, it runs extremely fast. about 3 times 
faster.

Is this because my test is generating a lot of garbage or more due to the GC 
process of GO having a big overhead by just running?

That is, would I still see a pef difference even in a system that does not 
allocate anything?

 

This test is ofcourse completely nonsense from a real world scenario, but 
still, it's an interesting challange trying to push the numbers down.

 

When profiling, I am not seeing anything specific that pops out at me. 

Anyone here up for giving a helping hand?

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go is two times slower than python code?

[Rich]

Have you tried searching https://godoc.org for a PCRE that may suite 
better? 
 https://godoc.org/github.com/glenn-brown/golang-pkg-pcre/src/pkg/pcre

On Tuesday, March 9, 2010 at 9:52:16 AM UTC-5, Alex Dong wrote:
>
> I'm writing a simple 'word picker' using golang and python. It
> basically reads a tweeter message from a csv file, tokenize it and put
> the word->id into a map called lexicon.  It turned out that same logic
> took 22 seconds for python 2.6 to finish whereas 57 seconds for
> golang!
>
> Wondering are there anything I've done wrong?
>
> Here is the python code:
> $ cat loader.py
> lexicon = {}
> pnct_ptn = re.compile(r'([\.,\\/\'"\?!=_\)\(\]\[\{\}:;]+|http://[^ ]
> +)')
> def tokenize(s):
> s = pnct_ptn.sub(' ', s)
> return [t for t in s.split() if len(t)>=3]
>
> for line in open("result.csv").readlines():
> parts = line.split(',', 3)
> if len(parts) != 4: continue
> msg = parts[3]
> s  = msg.decode('utf8','ignore').lower()
> for word in tokenize(s):
> if not lexicon.has_key(word):
> unique_words += 1
> lexicon[word] = unique_words
>
> Here is the go code:
> $ cat loader.go
> package main
>
> import (
> "bufio"
> "os"
> "regexp"
> "strings"
> )
>
> var (
> pr, _ = regexp.Compile(`(http://[^ ]+|['".\\,=()*:;?!/]|-)`)//
> pattern for removal
> )
>
> func tokenize(s string) []string {
> ms := pr.ReplaceAllString(strings.ToLower(s), " ")
> return strings.Split(ms, " ", 0)
> }
>
> func main() {
> lex := make(map[string] int)// lexicon
> dic := make(map[int] string)// lookup
> tw  := 0// total
> words
> ps  := false// present
>
> r, _ := os.Open("result.csv", os.O_RDONLY, 0444)
> defer r.Close()
> in := bufio.NewReader(r)
>
> for i := 0; i >= 0; i++ {
> line, err := in.ReadString('\n')
> if err != nil {
> break
> }
>
> parts:= strings.Split(line, ",", 4)
> if len(parts) != 4 {
> continue
> }
>
> ts := tokenize(parts[3])
> for d := 0; d < len(ts); d++ {
> w := ts[d]
> if len(w) < 3 {
> continue
> }
> _, ps = lex[w]
> if ps == false {
> lex[w] = tw
> dic[tw] = w
> tw ++
> }
> }
> }
> }
>
>
> Cheers,
> Alex
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Large 2D slice performance question

[Mandolyte]

Wow, this will take some time to digest. Regrettably I have required 
training today and won't be able to even play with this until tomorrow.

In my (parent,child), there are 304K unique parent values and 1.05M unique 
child values. Of course many child values are also parent values. Thus, 
total names are only just over 1.14M.

This data came from and will go back to a database and it must be correct 
CSV. And I know there is data that needs to be escaped per RFC 4180. Also, 
I had always thought that the CSV package was buffered since it has a 
Flush() method (and if not used, you will lose data!). At any rate, I'm 
reluctant to roll my own here since I've been bitten too many times by the 
data content.

Thanks for taking the time to review this!

On Thursday, December 1, 2016 at 4:21:42 AM UTC-5, Egon wrote:
>
> See whether this works better: 
> https://gist.github.com/egonelbre/d94ea561c3e63db009718e227e506b5b
>
> *There is a lot of room for improvements, (e.g. for your actual dataset 
> increase defaultNameCount).*
>
> *PS: Avoid csv package for writing files, use bufio and Write directly... 
> csv does some extra stuff, that you probably won't need.*
>
> btw. how many different names do you have?
>
> + Egon
>
> On Thursday, 1 December 2016 02:51:49 UTC+2, Mandolyte wrote:
>>
>> Thanks for the discussion! Package with tester is at:
>> https://github.com/mandolyte/TableRecursion
>>
>> While I can't share the data, I could take a sample set of paths for a 
>> root node, reverse engineer the pairs, and obfuscate... I've done this sort 
>> of thing before but it is a bit of work. So I'll treat that as a last 
>> resort. :-)
>>
>> On Wednesday, November 30, 2016 at 5:36:41 PM UTC-5, Mandolyte wrote:
>>>
>>> The finite set idea might work, but the set is well over 300K. The 
>>> strings (part numbers) are not regular.
>>>
>>> I could make a single pass over the "parent" column and record in a 
>>> map[string]int the index of the first occurrence. Then I would avoid 
>>> sort.Search() having to find it each time. Or use sort.Search() on the 
>>> smaller 300K slice if map performance was a problem...
>>>
>>> There is a lot of repetition in the "branches" - perhaps some caching 
>>> would be appropriate...
>>>
>>> thanks for the ideas!
>>>
>>> On Wednesday, November 30, 2016 at 9:31:56 AM UTC-5, adon...@google.com 
>>> wrote:

 On Wednesday, 30 November 2016 03:37:55 UTC+2, Mandolyte wrote:
>>
>> I have a fairly large 2D slice of strings, about 115M rows. These are 
>> (parent, child) pairs that, processed recursively, form a tree. I am 
>> "materializing" all possible trees as well as determining for each root 
>> node all child nodes for all levels for it.
>>
>  
 In addition to what Egon said:

 If the elements of the outer [][]string slice are all []string slices 
 of length 2, it would be much more efficient to use [2]string instead, 
 that 
 is, fixed-sized arrays of two strings, since this would avoid a lot of 
 memory allocation and pointer indirection.  The type of the outer slice 
 would become [][2]string.

 Also, if all your strings are drawn from a finite set, you'll find many 
 operations (such as comparison or set membership tests) much more 
 efficient 
 if you represent each string by its index in some side table.  Then 
 instead 
 of [2]string you can use [2]uint32, or perhaps even a narrower integer 
 type, which will make your main array more compact by at least a factor of 
 4.



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Help me beat Erlang, SkynetBenchmark challenge

[Roger Alsing]

There are other merits to actors, like supervision, distributed by default 
etc.
Different tools, different usecases, some overlap..


Roger


Den torsdag 1 december 2016 kl. 12:24:35 UTC+1 skrev Michael Jones:
>
> Here is an idea for you. 35% faster on my macbook pro:
>
> https://play.golang.org/p/3Mb5pR0V0J
>
>  
>
> Michael
>
>  
>
>  
>
> *From: * on behalf of Roger 
> Alsing 
> *Date: *Thursday, December 1, 2016 at 12:52 AM
> *To: *golang-nuts 
> *Subject: *[go-nuts] Help me beat Erlang, SkynetBenchmark challenge
>
>  
>
> I am currently trying to optimize the in process part of my Actor Model 
> framework for Go - GAM.
>
> In order to compare it to Erlang and Akka, I am using the "Skynet 
> Benchmark" https://github.com/atemerev/skynet#results-i7-4771-ubuntu-1510
>
>  
>
> I have ported this test to my lib: 
> https://github.com/AsynkronIT/gam/blob/dev/examples/spawnbenchmark/main.go
>
> It works and the performance is roughly the same as the Scala Akka 
> performace.
>
>  
>
> I have noticed that if I turn of GOGC, it runs extremely fast. about 3 
> times faster.
>
> Is this because my test is generating a lot of garbage or more due to the 
> GC process of GO having a big overhead by just running?
>
> That is, would I still see a pef difference even in a system that does not 
> allocate anything?
>
>  
>
> This test is ofcourse completely nonsense from a real world scenario, but 
> still, it's an interesting challange trying to push the numbers down.
>
>  
>
> When profiling, I am not seeing anything specific that pops out at me. 
>
> Anyone here up for giving a helping hand?
>
>  
>
>  
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Help me beat Erlang, SkynetBenchmark challenge

[Michael Jones]

Here is an idea for you. 35% faster on my macbook pro:

https://play.golang.org/p/3Mb5pR0V0J

 

Michael

 

 

From:  on behalf of Roger Alsing 

Date: Thursday, December 1, 2016 at 12:52 AM
To: golang-nuts 
Subject: [go-nuts] Help me beat Erlang, SkynetBenchmark challenge

 

I am currently trying to optimize the in process part of my Actor Model 
framework for Go - GAM.

In order to compare it to Erlang and Akka, I am using the "Skynet Benchmark" 
https://github.com/atemerev/skynet#results-i7-4771-ubuntu-1510

 

I have ported this test to my lib: 
https://github.com/AsynkronIT/gam/blob/dev/examples/spawnbenchmark/main.go

It works and the performance is roughly the same as the Scala Akka performace.

 

I have noticed that if I turn of GOGC, it runs extremely fast. about 3 times 
faster.

Is this because my test is generating a lot of garbage or more due to the GC 
process of GO having a big overhead by just running?

That is, would I still see a pef difference even in a system that does not 
allocate anything?

 

This test is ofcourse completely nonsense from a real world scenario, but 
still, it's an interesting challange trying to push the numbers down.

 

When profiling, I am not seeing anything specific that pops out at me. 

Anyone here up for giving a helping hand?

 

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] GOPATH default value in Go 1.8

[Tamás Gulácsi]

You can set it to whatever you want.
This is only a default - not too annoying as $HOME, and less hidden than 
.local/go

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] GOPATH default value in Go 1.8

[Peter Mogensen]

On 2016-12-01 11:18, Manlio Perillo wrote:

I have read the Go 1.8 release notes and I found that now GOPATH has a
default value.
This is a great thing: I have always I thought that GOPATH is one of the
weak point in the Go toolchain, both when I started programming in Go
and when I read the Go Programming language book.

However I think that the default value of $HOME/go is not the ideal one.
My settings are
GOPATH=$HOME/.local/lib/go:$HOME/src/go/
GOBIN=$HOME/.local/bin



A lot of people seem to have different usage patterns around GOPATH.
I happen to have my default GOPATH=~/go ... but I only use it for half 
of my projects.
The rest of my projects are self-contained and try to set GOPATH=`pwd` 
in the project root.

( where src/package/name is a symlink to the project root)

This allows for totally self contained projects, but some tools have 
problems with it because the "vendor" catalogue seems to work 
differently when building without a package name.

In other words:
"go build" fails to locate the vendor catalogue, but
"go build package/name" finds it.

There are ways around that,... and some of them involves relying on 
GOPATH not being set to any value initially for those projects.


So, I for one do not welcome our new GOPATH overlords.

Actually ... I would wish go tools wouldn't be so alien to having 
project-indiviual GOPATH.


/Peter

--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] GOPATH default value in Go 1.8

[Manlio Perillo]

I have read the Go 1.8 release notes and I found that now GOPATH has a 
default value.
This is a great thing: I have always I thought that GOPATH is one of the 
weak point in the Go toolchain, both when I started programming in Go and 
when I read the Go Programming language book.

However I think that the default value of $HOME/go is not the ideal one.
My settings are
GOPATH=$HOME/.local/lib/go:$HOME/src/go/
GOBIN=$HOME/.local/bin

Python also installs user installed modules in $HOME/.local/lib/pythonX.Y


Thanks
Manlio

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: locks and "happens before" withing a goroutine

[Daniel Fanjul]

> Within a single goroutine, the happens-before order is the order 
expressed by the program. 
If this is true, why is the order of 'a = 1; b = 2;' not ensured?
I think you may be forgetting that "compilers and processors may reorder 
the reads and writes executed within a single goroutine"

> Inside a single goroutine the code proceeds in statement order. 
The code generated by the compiler, right? Of course, it is typically the 
processor who reorders. But what is the limit of the reorder? Is 'a = 1; b 
= 2;' so simple that we worry about this case, but is 'a = 1; x.Lock(); b = 
2;' so complex that we assume that it will never be reordered? Now I see it 
is because we assume it is a memory barrier though it is undocumented. 
Thanks, Ian, I am sure now it is undocumented.

> - a = "hello, world" happens-before l.Unlock() 
That is false unless l.Unlock() is memory barrier or some other 
restriction. It happens it is a barrier, so you can say it truly happens 
before. The spec or the memory model does not mention it, so technically we 
cannot state it because of the spec or the memory model.

> If it didn't chaos would ensue, it would be as if statements in a 
function were executed in random order.
In theory, the statements are allowed to run in any order but only "when 
the reordering does not change the behavior within that goroutine as 
defined by the language specification", that is "when reads and writes must 
behave as if they executed in the order specified by the program". But I 
agree with you, Dave, in practice the deviation is small.

> I agree that this fact is not documented, but it is true nonetheless. 
Cool, thank you. I think it is quite important and really necessary to 
understand the memory model. Otherwise its examples do not make sense at 
all.

Thank you all, I finally and completely understood the memory model.


On Thursday, December 1, 2016 at 7:15:45 AM UTC+1, Ian Lance Taylor wrote:
>
> On Wed, Nov 30, 2016 at 5:44 PM, Daniel Fanjul 
>  wrote: 
> > I cannot tell if there is any memory barrier in the code of 
> > sync.RWMutex.Lock(), I don't understand it completely. I just think 
> there is 
> > not. 
> > https://golang.org/src/sync/mutex.go?s=1143:1165#L34 
>
> The sync/atomic routines provides memory barriers. 
>
> I agree that this fact is not documented, but it is true nonetheless. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Large 2D slice performance question

[Egon]

See whether this works better: 
https://gist.github.com/egonelbre/d94ea561c3e63db009718e227e506b5b

*There is a lot of room for improvements, (e.g. for your actual dataset 
increase defaultNameCount).*

*PS: Avoid csv package for writing files, use bufio and Write directly... 
csv does some extra stuff, that you probably won't need.*

btw. how many different names do you have?

+ Egon

On Thursday, 1 December 2016 02:51:49 UTC+2, Mandolyte wrote:
>
> Thanks for the discussion! Package with tester is at:
> https://github.com/mandolyte/TableRecursion
>
> While I can't share the data, I could take a sample set of paths for a 
> root node, reverse engineer the pairs, and obfuscate... I've done this sort 
> of thing before but it is a bit of work. So I'll treat that as a last 
> resort. :-)
>
> On Wednesday, November 30, 2016 at 5:36:41 PM UTC-5, Mandolyte wrote:
>>
>> The finite set idea might work, but the set is well over 300K. The 
>> strings (part numbers) are not regular.
>>
>> I could make a single pass over the "parent" column and record in a 
>> map[string]int the index of the first occurrence. Then I would avoid 
>> sort.Search() having to find it each time. Or use sort.Search() on the 
>> smaller 300K slice if map performance was a problem...
>>
>> There is a lot of repetition in the "branches" - perhaps some caching 
>> would be appropriate...
>>
>> thanks for the ideas!
>>
>> On Wednesday, November 30, 2016 at 9:31:56 AM UTC-5, adon...@google.com 
>> wrote:
>>>
>>> On Wednesday, 30 November 2016 03:37:55 UTC+2, Mandolyte wrote:
>
> I have a fairly large 2D slice of strings, about 115M rows. These are 
> (parent, child) pairs that, processed recursively, form a tree. I am 
> "materializing" all possible trees as well as determining for each root 
> node all child nodes for all levels for it.
>
  
>>> In addition to what Egon said:
>>>
>>> If the elements of the outer [][]string slice are all []string slices of 
>>> length 2, it would be much more efficient to use [2]string instead, that 
>>> is, fixed-sized arrays of two strings, since this would avoid a lot of 
>>> memory allocation and pointer indirection.  The type of the outer slice 
>>> would become [][2]string.
>>>
>>> Also, if all your strings are drawn from a finite set, you'll find many 
>>> operations (such as comparison or set membership tests) much more efficient 
>>> if you represent each string by its index in some side table.  Then instead 
>>> of [2]string you can use [2]uint32, or perhaps even a narrower integer 
>>> type, which will make your main array more compact by at least a factor of 
>>> 4.
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: synchronise read from two channels

[omarshariffdontlikeit]

Yeah I kinda figured I was coming to this wrong way - my tired brain had an 
unterminated loop. Thanks for the pointers! (all puns intended)

On Wednesday, November 30, 2016 at 5:08:06 PM UTC, omarsharif...@gmail.com 
wrote:
>
> Hi, I'm having a bit of a slow day... I'm trying to synchronise two reads 
> from two channels and can't get my fuzzy head round the problem.
>
> I have a channel containing urls (which will be feed periodically by a 
> named pipe by another program) and I have a cannel which consists of a pool 
> of available workers. I'd like to synchronise these and put a worker into a 
> third channel for active workers if there is a url in the channel and if 
> there is an available worker.
>
> Ideally I'd be able to do something like:
>
> go func(urlChan <-chan *url.URL, availableWorkerChan <-chan *worker, 
> activeWorkerChan chan<- *worker){
> for {
> select{
> case w := <-availableWorkerChan && w.Url = <-urlChan:
> activeWorkersChan <- w
> }
> }
> }(a, b, c)
>
> But obviously thats not right!
>
> Can anyone offer my tired brain a clue?
>
> Cheers!
> Ben
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.