Re: [go-nuts] weird sighup

[Ian Lance Taylor]

On Tue, May 19, 2020 at 7:53 PM Adrian Ratnapala
 wrote:
>
> I'd have expected that to work, because syntactically the slice is
> assigned to only after the new, smaller slice is fully constructed --
> including all the data copying.  But it still make me twitchy about
> the possibility that the runtime shortens the slice before reading all
> the data, resulting in an out-of-bounds read.
>
> As defensive programming, you can first make an explicit slice
> variable for the tail and then do the append.
>
> tail := s[i+1:]
> s = append(s[:i], tail...)
> // tail is still in scope ... so maybe that makes a differences?
>
> I'd be interested to see if this changes anything.  Because if it
> works, it smells like a bug in the runtime.

I'm really quite certain that that is not what is happening here.
This expression is a standard approach for removing an element from a
slice.  E.g., see https://golang.org/wiki/SliceTricks.

That said, it's impossible to tell what the problem is without a
complete standalone test case.

Ian




> On Wed, 20 May 2020 at 10:09, Ian Lance Taylor  wrote:
> >
> > On Tue, May 19, 2020 at 9:53 AM  wrote:
> > >
> > > I am using go-1.14.1. I am using this code to remove i-th element from 
> > > the slice.
> > >
> > > s = append(s[:i], s[i+1:]...)
> > >
> > > It works on my machine but gives panic on docker-container with error 
> > > "range out of bound" when it has just 1 element left.
> > >
> > > I want to check if this is an unsafe code and how can I improve this?
> >
> > It's hard to tell with only a code fragment.  The line you show will
> > crash if i >= len(s).
> >
> > Ian
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to golang-nuts+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUdMiJLvdfYe9%2B7fzjeuwJtu6uGxmW3UE-%3D%2BZ2DQLU%3D1Q%40mail.gmail.com.
>
>
>
> --
> Adrian Ratnapala

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXaDiJYJs8THTF7hA%2B9wqR07t31JA0eUBTTN4x1FJcmEg%40mail.gmail.com.

Re: [go-nuts] weird sighup

[Adrian Ratnapala]

I'd have expected that to work, because syntactically the slice is
assigned to only after the new, smaller slice is fully constructed --
including all the data copying.  But it still make me twitchy about
the possibility that the runtime shortens the slice before reading all
the data, resulting in an out-of-bounds read.

As defensive programming, you can first make an explicit slice
variable for the tail and then do the append.

tail := s[i+1:]
s = append(s[:i], tail...)
// tail is still in scope ... so maybe that makes a differences?

I'd be interested to see if this changes anything.  Because if it
works, it smells like a bug in the runtime.


On Wed, 20 May 2020 at 10:09, Ian Lance Taylor  wrote:
>
> On Tue, May 19, 2020 at 9:53 AM  wrote:
> >
> > I am using go-1.14.1. I am using this code to remove i-th element from the 
> > slice.
> >
> > s = append(s[:i], s[i+1:]...)
> >
> > It works on my machine but gives panic on docker-container with error 
> > "range out of bound" when it has just 1 element left.
> >
> > I want to check if this is an unsafe code and how can I improve this?
>
> It's hard to tell with only a code fragment.  The line you show will
> crash if i >= len(s).
>
> Ian
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUdMiJLvdfYe9%2B7fzjeuwJtu6uGxmW3UE-%3D%2BZ2DQLU%3D1Q%40mail.gmail.com.



-- 
Adrian Ratnapala

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAN%2BHj7jvdfx74d6gKctQNN-2jLKstoSpWnKauckLvGoEo23YDQ%40mail.gmail.com.

Re: [go-nuts] weird sighup

[Ian Lance Taylor]

On Tue, May 19, 2020 at 9:53 AM  wrote:
>
> I am using go-1.14.1. I am using this code to remove i-th element from the 
> slice.
>
> s = append(s[:i], s[i+1:]...)
>
> It works on my machine but gives panic on docker-container with error "range 
> out of bound" when it has just 1 element left.
>
> I want to check if this is an unsafe code and how can I improve this?

It's hard to tell with only a code fragment.  The line you show will
crash if i >= len(s).

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUdMiJLvdfYe9%2B7fzjeuwJtu6uGxmW3UE-%3D%2BZ2DQLU%3D1Q%40mail.gmail.com.

Re: [go-nuts] DNS lookup timeouts could return net.DNSError instead of poll.TimeoutError

[Ian Lance Taylor]

On Tue, May 19, 2020 at 4:27 PM  wrote:
>
> I get DNS lookup timeouts many times a day when running Go HTTP clients on 
> Kubernetes, where UDP packets are sometimes dropped due to a race condition 
> in the Linux kernel. I'd like to propose a small standard library change to 
> help debug these scenarios and get some feedback on the idea and 
> implementation here before posting it as a Github issue.
>
> For any http.Client use with a timeout set on the transport, when the timeout 
> is exceeded and an error is returned it can be challenging to discover the 
> cause of the timeout from the error message. Is it DNS related or TCP related 
> or something else? Consider the following simulation of a DNS lookup that 
> does not return within the timeout.
>
> package main
>
> import (
> "context"
> "net"
> "net/http"
> "time"
> )
>
> var timeout = 50 * time.Millisecond
> var host = "http://example.com/;
>
> // delayedDialer introduces a delay when performing DNS lookups.
> func delayedDialer(ctx context.Context, network, address string) (net.Conn, 
> error) {
> time.Sleep(2 * timeout)
> d := net.Dialer{}
> return d.DialContext(ctx, network, address)
> }
>
> func main() {
> r := {
> PreferGo: true,
> Dial: delayedDialer,
> }
> tr := {
> DialContext: ({
> Timeout:  timeout,
> Resolver: r,
> }).DialContext,
> }
> client := {Transport: tr}
> _, err := client.Get(host)
> if err != nil {
> panic(err)
> }
> }
>
> In the event of a DNS lookup timeout, the output is:
>
> panic: Get "http://example.com/": dial tcp: i/o timeout
>
> In the event of a TCP connection timeout (comment out the time.Sleep() call) 
> the output is:
>
> panic: Get "http://example.com/": dial tcp 93.184.216.34:80: i/o timeout
>
> The missing IP address in the DNS lookup timeout output is the only clue that 
> the root cause is a DNS problem, rather than a TCP problem. If users do not 
> expect to see an IP address in that error message, they won't know it is 
> missing, and their debugging and testing effort may be wasted investigating 
> possible TCP-related causes, instead of investigating DNS (which normally 
> runs over UDP).
>
> The error message could communicate the root cause to users more clearly. 
> After a DNS lookup timeout, the error returned by client.Get() looks like 
> this in Go 1.14.3:
>
> {
>   Op:  "Get",
>   URL: "http://example.com/;,
>   Err: {
> Op: "dial",
> Net:"tcp",
> Source: nil,
> Addr:   nil,
> Err:{},
>   },
> }
>
> Note: The {} will become {} in future 
> releases after commit d422f54.
>
> The resolver in net/lookup.go is the source of the *poll.TimeoutError. 
> Instead it could return a *net.DNSError error when a lookup times out. Doing 
> so would provide additional error context for all use cases (not only 
> http.Client) and may save debugging time. For example client.Get() could 
> return:
>
> {
>   Op:  "Get",
>   URL: "http://example.com/;,
>   Err: {
> Op: "dial",
> Net:"tcp",
> Source: nil,
> Addr:   nil,
> Err:{
>   Err: "i/o timeout",
>   Name:"example.com",
>   Server:  "",
>   IsTimeout:   true,
>   IsTemporary: false,
>   IsNotFound:  false,
> },
>   },
> }
>
> This would preserve the existing "i/o timeout" string and the ability to call 
> err.Timeout(), while adding "lookup example.com: " to the Error() output and 
> the capability to explicitly check for a *net.DNSError. The output from my 
> simulator code would also be more explicit:
>
> panic: Get "http://example.com/": dial tcp: lookup example.com: i/o timeout
>
> The following change to implement this works in my usage and passes the tests 
> in all.bash.
>
> $ git diff
> diff --git a/src/net/lookup.go b/src/net/lookup.go
> index 5f7119872a..b1a22dd4e6 100644
> --- a/src/net/lookup.go
> +++ b/src/net/lookup.go
> @@ -314,6 +314,9 @@ func (r *Resolver) lookupIPAddr(ctx context.Context, 
> network, host string) ([]IP
> }()
> }
> err := mapErr(ctx.Err())
> +   if t, ok := err.(timeout); ok && t.Timeout() {
> +   err = {Err: err.Error(), Name: host, 
> IsTimeout: true}
> +   }
> if trace != nil && trace.DNSDone != nil {
> trace.DNSDone(nil, false, err)
> }
>
> I can submit a Github issue and pull request if anyone thinks that this is a 
> good idea and implementation.

I don't know if that is the right implementation, but what you say
sounds plausible, and I encourage you to open an issue.  Thanks.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the 

[go-nuts] DNS lookup timeouts could return net.DNSError instead of poll.TimeoutError

[therealmarkhale]

Hi,

I get DNS lookup timeouts many times a day when running Go HTTP clients on 
Kubernetes, where UDP packets are sometimes dropped due to a race condition 
in the Linux kernel. I'd like to propose a small standard library change to 
help debug these scenarios and get some feedback on the idea and 
implementation here before posting it as a Github issue.

For any http.Client use with a timeout set on the transport, when the 
timeout is exceeded and an error is returned it can be challenging to 
discover the cause of the timeout from the error message. Is it DNS related 
or TCP related or something else? Consider the following simulation of a 
DNS lookup that does not return within the timeout.

package main
  
import (
"context"
"net"
"net/http"
"time"
)

var timeout = 50 * time.Millisecond
var host = "http://example.com/;

// delayedDialer introduces a delay when performing DNS lookups.
func delayedDialer(ctx context.Context, network, address string) (net.Conn, 
error) {
time.Sleep(2 * timeout)
d := net.Dialer{}
return d.DialContext(ctx, network, address)
}

func main() {
r := {
PreferGo: true,
Dial: delayedDialer,
}
tr := {
DialContext: ({
Timeout:  timeout,
Resolver: r,
}).DialContext,
}
client := {Transport: tr}
_, err := client.Get(host)
if err != nil {
panic(err)
}
}

In the event of a DNS lookup timeout, the output is:

panic: Get "http://example.com/": dial tcp: i/o timeout

In the event of a TCP connection timeout (comment out the time.Sleep() 
call) the output is:

panic: Get "http://example.com/": dial tcp 93.184.216.34:80: i/o timeout

The missing IP address in the DNS lookup timeout output is the only clue 
that the root cause is a DNS problem, rather than a TCP problem. If users 
do not expect to see an IP address in that error message, they won't know 
it is missing, and their debugging and testing effort may be wasted 
investigating possible TCP-related causes, instead of investigating DNS 
(which normally runs over UDP).

The error message could communicate the root cause to users more 
clearly. After a DNS lookup timeout, the error returned by client.Get() 
looks like this in Go 1.14.3:

{
  Op:  "Get",
  URL: "http://example.com/;,
  Err: {
Op: "dial",
Net:"tcp",
Source: nil,
Addr:   nil,
Err:{},
  },
}

Note: The {} will become {} in future 
releases after commit d422f54 

.

The resolver in net/lookup.go 
 is the 
source of the *poll.TimeoutError. Instead it could return a *net.DNSError 
error when a lookup times out. Doing so would provide additional error 
context for all use cases (not only http.Client) and may save debugging 
time. For example client.Get() could return:

{
  Op:  "Get",
  URL: "http://example.com/;,
  Err: {
Op: "dial",
Net:"tcp",
Source: nil,
Addr:   nil,
Err:{
  Err: "i/o timeout",
  Name:"example.com",
  Server:  "",
  IsTimeout:   true,
  IsTemporary: false,
  IsNotFound:  false,
},
  },
}

This would preserve the existing "i/o timeout" string and the ability to 
call err.Timeout(), while adding "lookup example.com: " to the Error() 
output and the capability to explicitly check for a *net.DNSError. The 
output from my simulator code would also be more explicit:

panic: Get "http://example.com/": dial tcp: lookup example.com: i/o timeout

The following change to implement this works in my usage and passes the 
tests in all.bash.

$ git diff
diff --git a/src/net/lookup.go b/src/net/lookup.go
index 5f7119872a..b1a22dd4e6 100644
--- a/src/net/lookup.go
+++ b/src/net/lookup.go
@@ -314,6 +314,9 @@ func (r *Resolver) lookupIPAddr(ctx context.Context, 
network, host string) ([]IP
}()
}
err := mapErr(ctx.Err())
+   if t, ok := err.(timeout); ok && t.Timeout() {
+   err = {Err: err.Error(), Name: host, 
IsTimeout: true}
+   }
if trace != nil && trace.DNSDone != nil {
trace.DNSDone(nil, false, err)
}

I can submit a Github issue and pull request if anyone thinks that this is 
a good idea and implementation.

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/26d8e190-107e-4dd1-9de0-cab02fe5699a%40googlegroups.com.

[go-nuts] Re: help understanding weird Benchmark results

[mike]

As Michael Jones said, you still need to play by the testing package's 
benchmark rules for it to be able to benchmark your code.

So something along these lines.

func BenchmarkMarshalSample(b *testing.B) {
for i:=0; i < b.N; i++ {
var sum int64
// start := time.Now()
for i := 0; i < 10_000_000; i++ {
sum += rand.Int63n(0x)
}
// b.Logf("Sum %e Duration %v", float64(sum), time.Now().Sub(start))
}
}


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/36e51c75-6571-41df-b829-038d284c44bd%40googlegroups.com.

[go-nuts] Re: Client certificate not sent to server

[Mark Petrovic]

I know this is an old thread, but I found it, so others might, too.  

I struggled with a similar problem of the Go HTTP client not sending client 
certs when challenged by the server.  Yet, curl worked fine when given the 
same client certs, calling the same server endpoint.  The cert chains I was 
dealing with look like

leaf + intermediate + root


The solution for me was to provide the server (Envoy proxy) with a the root 
cert AND intermediate cert, the latter of which signed my Go client cert.  
When the server challenges the client for its cert, the server will send a 
TLS Certificate Request with, among other things, the certificate authority 
that must have signed the client cert.  Without including the intermediate 
cert with the root cert on the server side, the client cannot not build a 
complete path from its client cert to the root and therefore cannot 
determine which cert to send in response.  The client will not use 
tls.Config.RootCAs in this analysis (RootCAs is for validating the server 
cert chain).  Moreover, when the client parses its cert with 

cert, err := tls.LoadX509KeyPair(*certFile, *keyFile)


only the leaf is returned even if the certFile contains the concatenated 
intermediate.

Including the full cert bundle on the server side solves this for me.

Hope that helps.



On Tuesday, March 17, 2015 at 10:19:01 AM UTC-7, Dots Connected wrote:
>
> Hello!
>
> I am trying to access an API served by nginx (with a self-signed SSL cert) 
> that requires client certificate authentication, and while I can 
> successfully auth with curl, I unable to accomplish the same with golang.
>
> Example using curl:
>
> # Will not work
> curl -k https://127.0.0.1/api
>
> # Works
> curl -k -E example_key+cert.pem https://127.0.0.1/api
>
>
> I found an example from this list that I've tried to adapt:
>
> https://groups.google.com/forum/#!topic/golang-nuts/dEfqPOSccIchttps://groups.google.com/forum/#!topic/golang-nuts/dEfqPOSccIc
>  
> 
> And another from github:
> https://gist.github.com/michaljemala/d6f4e01c4834bf47a9c4
>
> Despite this, golang gets the same result as the first curl with no client 
> cert.
> Am I missing something, or is this a regression in go?
>
> I'm on go version go1.4.2 linux/amd64. My code (or via 
> http://pastebin.com/LGiXZpgx):
>
> package main
>
> import (
> "crypto/tls"
> "io/ioutil"
> "log"
> "net/http"
> )
>
> func main() {
>
> certFile := "example_cert.pem"
> keyFile := "example_key.pem"
>
> // Load client cert
> cert, err := tls.LoadX509KeyPair(certFile, keyFile)
> if err != nil {
> log.Fatal(err)
> }
>
> // Setup HTTPS client
> tlsConfig := {
> Certificates: []tls.Certificate{cert},
> ClientAuth: tls.RequireAndVerifyClientCert,
> InsecureSkipVerify: true,
> }
> tlsConfig.BuildNameToCertificate()
>
> transport := {
> TLSClientConfig: tlsConfig,
> }
> client := {
> Transport: transport,
> }
>
> // Do GET something
> resp, err := client.Get("https://localhost/api/;)
> if err != nil {
> log.Fatal(err)
> }
> defer resp.Body.Close()
>
> // Dump response
> data, err := ioutil.ReadAll(resp.Body)
> if err != nil {
> log.Fatal(err)
> }
> log.Println(string(data))
> }
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/43ca36e6-bffe-4b2f-8ee6-c34288685023%40googlegroups.com.

Re: [go-nuts] golang protobuf, struct copy problem

[Vasiliy Tolstov]

пт, 8 мая 2020 г. в 22:22, Robert Engels :
>
> Have the compute struct contain the proto struct.
>

I think that question is more about how to copy fields from one struct
to another. For example one struct contains some tags (for example to
work with some db orm), and proto struct don't have such tags.

> On May 8, 2020, at 1:56 PM, cheng dong  wrote:
>
> 
> i use protobuf to do rpc in golang . for every message XXX in proto,  i have 
> to define a struct XXX in go, compute and then copy its field one by one to 
> pb.XXX（because struct XXX have some field not in pb.XXX), when i have many 
> struct XXX, it is very ineffective and very slow.
>
> so how could i use a single struct both used for compute and marshal ?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/f67468e5-cb3b-4026-903a-f7ac49402bbd%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/71BBB1B9-9CDB-4E46-A486-A1FC7BF362BC%40ix.netcom.com.



-- 
Vasiliy Tolstov,
e-mail: v.tols...@selfip.ru

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CACaajQsr06xYbv1N1Hg2Ax2KOTPeEa%2BoMowQSVdNOVgYie9kHg%40mail.gmail.com.

[go-nuts] Re: Sscanf() skip elements

['simon place' via golang-nuts]

obviously, you could do this... https://play.golang.org/p/isqA3N0LBkn

or better?...  https://play.golang.org/p/8hXgZqBNyns

On Friday, 10 April 2020 18:17:53 UTC+1, Daniel Gorbe wrote:
>
> Hi!
>
>
> It is possible to skip elements of a string with Sscanf()? As in C with 
> *%*s.*
>
> Example:
>
> *var res string*
>
> *text := "A simple text"*
>
> *fmt.Sscanf(text, "%*s %*s %s", )*
>
> *fmt.Printf("%s\n", res) // print: "text"*
>
>
> Best regards,
>
> Daniel
>
> -- 
> Daniel Gorbe
> GNU/Linux System Administrator - IT Security Specialist
> PGP: 0x5AEB729B3E642A16
> Web: https://danielgorbe.com
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e053bcc6-ea93-41ef-a41b-eeab920029ea%40googlegroups.com.

Re: [go-nuts] Re: help understanding weird Benchmark results

[Marvin Renich]

* Warren Bare  [200519 13:53]:
> OK, I added a sum of the rand to the demo code and the results are the 
> same.  Since it is displaying the sum, it seems clear that the code is not 
> optimized away.
> 
> Again, I am NOT trying to time each iteration of the loop.  This is a 
> minimal demonstration of a weirdness I was seeing in my own benchmarks.  
> Just pretend the loop is one chunk of code that takes time and is not 
> optimized away.
> 
> Any other ideas?
> 
> func BenchmarkMarshalSample(b *testing.B) {
> var sum int64
> start := time.Now()
> for i := 0; i < 10_000_000; i++ {
> sum += rand.Int63n(0x)
> }
> b.Logf("Sum %e Duration %v", float64(sum), time.Now().Sub(start))
> }

You do not clearly understand how benchmarking in the testing package
works.  In order to get meaningful results, your benchmark function
_must_ iterate b.N times.  The benchmark framework is going to run your
function several times, with different values of b.N until it can get a
meaningful time, and then it does the math to give you "per iteration"
times.  Read carefully the section "Benchmarks" under
https://golang.org/pkg/testing/.

For the above case (given that you are not trying to time one
invocation of rand.Int63n, but a fixed number) you would write your code
like this:

func BenchmarkMarshalSample(b *testing.B) {
var sum int64
start := time.Now()
for j := 0; j < b.N; j++ {
for i := 0; i < 10_000_000; i++ {
sum += rand.Int63n(0x)
}
}
b.Logf("Sum %e Duration %v", float64(sum), time.Now().Sub(start))
}

The benchmark results will give the time for _one_ iteration of the
_outer_ loop, so the "op" in "ns/op" will be 10_000_000 iterations of
the inner loop.

...Marvin

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20200519181906.5xvifyyrowbnfzrs%40basil.wdw.

[go-nuts] Re: help understanding weird Benchmark results

[Warren Bare]

OK, I got it.  You were trying to change the code I was benchmarking, and I 
did not realize you were trying to say a benchmark must have the ability to 
run b.N times.

Yes, that was the problem.  This fixed it:

func BenchmarkMarshalSample(b *testing.B) {
var sum int64
for i:= 0; i < b.N; i++ {
start := time.Now()
for x := 0; x < 10_000_000; x++ {
sum += rand.Int63n(0x)
}
b.Logf("Sum %e Duration %v", float64(sum), time.Now().Sub(start))
}
}


On Tuesday, May 19, 2020 at 1:53:19 PM UTC-4, Warren Bare wrote:
>
> OK, I added a sum of the rand to the demo code and the results are the 
> same.  Since it is displaying the sum, it seems clear that the code is not 
> optimized away.
>
> Again, I am NOT trying to time each iteration of the loop.  This is a 
> minimal demonstration of a weirdness I was seeing in my own benchmarks.  
> Just pretend the loop is one chunk of code that takes time and is not 
> optimized away.
>
> Any other ideas?
>
> func BenchmarkMarshalSample(b *testing.B) {
> var sum int64
> start := time.Now()
> for i := 0; i < 10_000_000; i++ {
> sum += rand.Int63n(0x)
> }
> b.Logf("Sum %e Duration %v", float64(sum), time.Now().Sub(start))
> }
>
>
>
> On Tuesday, May 19, 2020 at 12:20:43 PM UTC-4, Warren Bare wrote:
>>
>> Hi Folks,
>>
>> I'm getting weird results from Benchmark.  Maybe someone can help me 
>> understand this.  I'm running on amd-64 (Threadripper 16 core 32 thread) 
>> Windows 10.  Go 1.14.3
>>
>> I have the benchmark below (main_test.go) on a minimum "hello world" 
>> main.go (just like playground).
>>
>> When I run the benchmark as it is below, I get the results included just 
>> below here.  Notice it reports 0.135 ns/op but the time is actually 135 
>> *ms* so it is off by a factor of 1 billion.  It is like it trying to 
>> report in seconds but did not change the label from ns to s.
>>
>> Further, if I increase the loop 10x from 10_000_000 to 100_000_000, then 
>> it prints Duration 1.349 seconds (good) and now the Benchmark time has 
>> increased by a factor of 10 *billion *and is now correctly reported as 
>> 1349224200 ns/op
>>
>> What am I missing here?
>>
>>
>> BenchmarkMarshalSample-32   10   0.135 ns/op   0 
>> B/op  0 allocs/op
>> --- BENCH: BenchmarkMarshalSample-32
>> main_test.go:14: Duration 136.1221ms
>> main_test.go:14: Duration 135.1214ms
>> main_test.go:14: Duration 134.1763ms
>> main_test.go:14: Duration 135.1217ms
>> main_test.go:14: Duration 135.1298ms
>> main_test.go:14: Duration 135.1217ms
>> main_test.go:14: Duration 135.1218ms
>> main_test.go:14: Duration 135.1213ms
>> main_test.go:14: Duration 135.1298ms
>> main_test.go:14: Duration 135.1216ms
>> ... [output truncated]
>> PASS
>>
>>
>>
>> package main
>>
>> import (
>> "math/rand"
>> "testing"
>> "time"
>> )
>>
>> func BenchmarkMarshalSample(b *testing.B) {
>> start := time.Now()
>> for i := 0; i < 10_000_000; i++ {
>> rand.Int63()
>> }
>> b.Log("Duration", time.Now().Sub(start))
>> }
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1f6e43a6-3c62-4c0f-99af-f277ba8cba55%40googlegroups.com.

[go-nuts] Re: help understanding weird Benchmark results

[Warren Bare]

OK, I added a sum of the rand to the demo code and the results are the 
same.  Since it is displaying the sum, it seems clear that the code is not 
optimized away.

Again, I am NOT trying to time each iteration of the loop.  This is a 
minimal demonstration of a weirdness I was seeing in my own benchmarks.  
Just pretend the loop is one chunk of code that takes time and is not 
optimized away.

Any other ideas?

func BenchmarkMarshalSample(b *testing.B) {
var sum int64
start := time.Now()
for i := 0; i < 10_000_000; i++ {
sum += rand.Int63n(0x)
}
b.Logf("Sum %e Duration %v", float64(sum), time.Now().Sub(start))
}



On Tuesday, May 19, 2020 at 12:20:43 PM UTC-4, Warren Bare wrote:
>
> Hi Folks,
>
> I'm getting weird results from Benchmark.  Maybe someone can help me 
> understand this.  I'm running on amd-64 (Threadripper 16 core 32 thread) 
> Windows 10.  Go 1.14.3
>
> I have the benchmark below (main_test.go) on a minimum "hello world" 
> main.go (just like playground).
>
> When I run the benchmark as it is below, I get the results included just 
> below here.  Notice it reports 0.135 ns/op but the time is actually 135 
> *ms* so it is off by a factor of 1 billion.  It is like it trying to 
> report in seconds but did not change the label from ns to s.
>
> Further, if I increase the loop 10x from 10_000_000 to 100_000_000, then 
> it prints Duration 1.349 seconds (good) and now the Benchmark time has 
> increased by a factor of 10 *billion *and is now correctly reported as 
> 1349224200 ns/op
>
> What am I missing here?
>
>
> BenchmarkMarshalSample-32   10   0.135 ns/op   0 
> B/op  0 allocs/op
> --- BENCH: BenchmarkMarshalSample-32
> main_test.go:14: Duration 136.1221ms
> main_test.go:14: Duration 135.1214ms
> main_test.go:14: Duration 134.1763ms
> main_test.go:14: Duration 135.1217ms
> main_test.go:14: Duration 135.1298ms
> main_test.go:14: Duration 135.1217ms
> main_test.go:14: Duration 135.1218ms
> main_test.go:14: Duration 135.1213ms
> main_test.go:14: Duration 135.1298ms
> main_test.go:14: Duration 135.1216ms
> ... [output truncated]
> PASS
>
>
>
> package main
>
> import (
> "math/rand"
> "testing"
> "time"
> )
>
> func BenchmarkMarshalSample(b *testing.B) {
> start := time.Now()
> for i := 0; i < 10_000_000; i++ {
> rand.Int63()
> }
> b.Log("Duration", time.Now().Sub(start))
> }
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c0775f1f-9eeb-4ac4-8615-027caa02f1ed%40googlegroups.com.

Re: [go-nuts] Re: help understanding weird Benchmark results

[Michael Jones]

as was explained, the loop needs to be "for i:=0; i < b.N; i++"

as was mentioned, the compiler's dead code elimination efforts can
frustrate benchmarking. they way to make sure the test code survives is to
not let it be dead code. for example

// external var dummy

func
for i:=0; i < b.N; i++ {
  dummy += rand.Int63()
}

On Tue, May 19, 2020 at 10:22 AM Warren Bare  wrote:

> >You are supposed to run the loop b.N times, not some fixed constant.
>
> I understand.  This is a simulation of a single bigger task that takes a
> while.  I'm not trying to time the rand function inside the loop.  The loop
> is simply to burn time.  This simple function is a minimal example that
> demonstrates a problem I was having with my own real benchmark test.
>
> I don't know for sure that the compiler is NOT optimizing away this rand
> function, but I can tell you this problem was happening on my own code that
> can not be optimized away.
>
> Thanks.
>
>
>
> On Tuesday, May 19, 2020 at 12:39:33 PM UTC-4, Volker Dobler wrote:
>>
>> You are supposed to run the loop b.N times, not
>> some fixed constant. Also make sure the compiler
>> doesn't optimize away the whole function.
>>
>> V.
>>
>> On Tuesday, 19 May 2020 18:20:43 UTC+2, Warren Bare wrote:
>>>
>>> Hi Folks,
>>>
>>> I'm getting weird results from Benchmark.  Maybe someone can help me
>>> understand this.  I'm running on amd-64 (Threadripper 16 core 32 thread)
>>> Windows 10.  Go 1.14.3
>>>
>>> I have the benchmark below (main_test.go) on a minimum "hello world"
>>> main.go (just like playground).
>>>
>>> When I run the benchmark as it is below, I get the results included just
>>> below here.  Notice it reports 0.135 ns/op but the time is actually 135
>>> *ms* so it is off by a factor of 1 billion.  It is like it trying to
>>> report in seconds but did not change the label from ns to s.
>>>
>>> Further, if I increase the loop 10x from 10_000_000 to 100_000_000, then
>>> it prints Duration 1.349 seconds (good) and now the Benchmark time has
>>> increased by a factor of 10 *billion *and is now correctly reported as
>>> 1349224200 ns/op
>>>
>>> What am I missing here?
>>>
>>>
>>> BenchmarkMarshalSample-32   10   0.135 ns/op   
>>> 0 B/op  0 allocs/op
>>> --- BENCH: BenchmarkMarshalSample-32
>>> main_test.go:14: Duration 136.1221ms
>>> main_test.go:14: Duration 135.1214ms
>>> main_test.go:14: Duration 134.1763ms
>>> main_test.go:14: Duration 135.1217ms
>>> main_test.go:14: Duration 135.1298ms
>>> main_test.go:14: Duration 135.1217ms
>>> main_test.go:14: Duration 135.1218ms
>>> main_test.go:14: Duration 135.1213ms
>>> main_test.go:14: Duration 135.1298ms
>>> main_test.go:14: Duration 135.1216ms
>>> ... [output truncated]
>>> PASS
>>>
>>>
>>>
>>> package main
>>>
>>> import (
>>> "math/rand"
>>> "testing"
>>> "time"
>>> )
>>>
>>> func BenchmarkMarshalSample(b *testing.B) {
>>> start := time.Now()
>>> for i := 0; i < 10_000_000; i++ {
>>> rand.Int63()
>>> }
>>> b.Log("Duration", time.Now().Sub(start))
>>> }
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/2abb7390-3c3c-4172-aead-021dbf500bad%40googlegroups.com
> 
> .
>


-- 

*Michael T. jonesmichael.jo...@gmail.com *

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALoEmQwN%2Bou8hqYEi5R1RuC%2Bdvbzzemsha35QA5LwK87Q_whVA%40mail.gmail.com.

[go-nuts] Re: help understanding weird Benchmark results

[Warren Bare]

>You are supposed to run the loop b.N times, not some fixed constant. 

I understand.  This is a simulation of a single bigger task that takes a 
while.  I'm not trying to time the rand function inside the loop.  The loop 
is simply to burn time.  This simple function is a minimal example that 
demonstrates a problem I was having with my own real benchmark test.

I don't know for sure that the compiler is NOT optimizing away this rand 
function, but I can tell you this problem was happening on my own code that 
can not be optimized away.  

Thanks.



On Tuesday, May 19, 2020 at 12:39:33 PM UTC-4, Volker Dobler wrote:
>
> You are supposed to run the loop b.N times, not
> some fixed constant. Also make sure the compiler
> doesn't optimize away the whole function.
>
> V.
>
> On Tuesday, 19 May 2020 18:20:43 UTC+2, Warren Bare wrote:
>>
>> Hi Folks,
>>
>> I'm getting weird results from Benchmark.  Maybe someone can help me 
>> understand this.  I'm running on amd-64 (Threadripper 16 core 32 thread) 
>> Windows 10.  Go 1.14.3
>>
>> I have the benchmark below (main_test.go) on a minimum "hello world" 
>> main.go (just like playground).
>>
>> When I run the benchmark as it is below, I get the results included just 
>> below here.  Notice it reports 0.135 ns/op but the time is actually 135 
>> *ms* so it is off by a factor of 1 billion.  It is like it trying to 
>> report in seconds but did not change the label from ns to s.
>>
>> Further, if I increase the loop 10x from 10_000_000 to 100_000_000, then 
>> it prints Duration 1.349 seconds (good) and now the Benchmark time has 
>> increased by a factor of 10 *billion *and is now correctly reported as 
>> 1349224200 ns/op
>>
>> What am I missing here?
>>
>>
>> BenchmarkMarshalSample-32   10   0.135 ns/op   0 
>> B/op  0 allocs/op
>> --- BENCH: BenchmarkMarshalSample-32
>> main_test.go:14: Duration 136.1221ms
>> main_test.go:14: Duration 135.1214ms
>> main_test.go:14: Duration 134.1763ms
>> main_test.go:14: Duration 135.1217ms
>> main_test.go:14: Duration 135.1298ms
>> main_test.go:14: Duration 135.1217ms
>> main_test.go:14: Duration 135.1218ms
>> main_test.go:14: Duration 135.1213ms
>> main_test.go:14: Duration 135.1298ms
>> main_test.go:14: Duration 135.1216ms
>> ... [output truncated]
>> PASS
>>
>>
>>
>> package main
>>
>> import (
>> "math/rand"
>> "testing"
>> "time"
>> )
>>
>> func BenchmarkMarshalSample(b *testing.B) {
>> start := time.Now()
>> for i := 0; i < 10_000_000; i++ {
>> rand.Int63()
>> }
>> b.Log("Duration", time.Now().Sub(start))
>> }
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2abb7390-3c3c-4172-aead-021dbf500bad%40googlegroups.com.

[go-nuts] weird sighup

[anderson . mill . road]

Hi, 

I am using go-1.14.1. I am using this code to remove i-th element from the 
slice.

s = append(s[:i], s[i+1:]...) 

It works on my machine but gives panic on docker-container with error 
"range out of bound" when it has just 1 element left. 

I want to check if this is an unsafe code and how can I improve this?

Thanks,

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/af5d785e-a93a-44e5-afdf-93056966c4e7%40googlegroups.com.

[go-nuts] Re: help understanding weird Benchmark results

[Volker Dobler]

You are supposed to run the loop b.N times, not
some fixed constant. Also make sure the compiler
doesn't optimize away the whole function.

V.

On Tuesday, 19 May 2020 18:20:43 UTC+2, Warren Bare wrote:
>
> Hi Folks,
>
> I'm getting weird results from Benchmark.  Maybe someone can help me 
> understand this.  I'm running on amd-64 (Threadripper 16 core 32 thread) 
> Windows 10.  Go 1.14.3
>
> I have the benchmark below (main_test.go) on a minimum "hello world" 
> main.go (just like playground).
>
> When I run the benchmark as it is below, I get the results included just 
> below here.  Notice it reports 0.135 ns/op but the time is actually 135 
> *ms* so it is off by a factor of 1 billion.  It is like it trying to 
> report in seconds but did not change the label from ns to s.
>
> Further, if I increase the loop 10x from 10_000_000 to 100_000_000, then 
> it prints Duration 1.349 seconds (good) and now the Benchmark time has 
> increased by a factor of 10 *billion *and is now correctly reported as 
> 1349224200 ns/op
>
> What am I missing here?
>
>
> BenchmarkMarshalSample-32   10   0.135 ns/op   0 
> B/op  0 allocs/op
> --- BENCH: BenchmarkMarshalSample-32
> main_test.go:14: Duration 136.1221ms
> main_test.go:14: Duration 135.1214ms
> main_test.go:14: Duration 134.1763ms
> main_test.go:14: Duration 135.1217ms
> main_test.go:14: Duration 135.1298ms
> main_test.go:14: Duration 135.1217ms
> main_test.go:14: Duration 135.1218ms
> main_test.go:14: Duration 135.1213ms
> main_test.go:14: Duration 135.1298ms
> main_test.go:14: Duration 135.1216ms
> ... [output truncated]
> PASS
>
>
>
> package main
>
> import (
> "math/rand"
> "testing"
> "time"
> )
>
> func BenchmarkMarshalSample(b *testing.B) {
> start := time.Now()
> for i := 0; i < 10_000_000; i++ {
> rand.Int63()
> }
> b.Log("Duration", time.Now().Sub(start))
> }
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5a4f710b-4d47-4f2f-9886-ed9cfc897de1%40googlegroups.com.

[go-nuts] help understanding weird Benchmark results

[Warren Bare]

Hi Folks,

I'm getting weird results from Benchmark.  Maybe someone can help me 
understand this.  I'm running on amd-64 (Threadripper 16 core 32 thread) 
Windows 10.  Go 1.14.3

I have the benchmark below (main_test.go) on a minimum "hello world" 
main.go (just like playground).

When I run the benchmark as it is below, I get the results included just 
below here.  Notice it reports 0.135 ns/op but the time is actually 135 *ms* so 
it is off by a factor of 1 billion.  It is like it trying to report in 
seconds but did not change the label from ns to s.

Further, if I increase the loop 10x from 10_000_000 to 100_000_000, then it 
prints Duration 1.349 seconds (good) and now the Benchmark time has 
increased by a factor of 10 *billion *and is now correctly reported as 
1349224200 ns/op

What am I missing here?

BenchmarkMarshalSample-32   10   0.135 ns/op   0 
B/op  0 allocs/op
--- BENCH: BenchmarkMarshalSample-32
main_test.go:14: Duration 136.1221ms
main_test.go:14: Duration 135.1214ms
main_test.go:14: Duration 134.1763ms
main_test.go:14: Duration 135.1217ms
main_test.go:14: Duration 135.1298ms
main_test.go:14: Duration 135.1217ms
main_test.go:14: Duration 135.1218ms
main_test.go:14: Duration 135.1213ms
main_test.go:14: Duration 135.1298ms
main_test.go:14: Duration 135.1216ms
... [output truncated]
PASS



package main

import (
"math/rand"
"testing"
"time"
)

func BenchmarkMarshalSample(b *testing.B) {
start := time.Now()
for i := 0; i < 10_000_000; i++ {
rand.Int63()
}
b.Log("Duration", time.Now().Sub(start))
}

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/33c4ab46-18ae-46c2-99e9-e467d6f3e184%40googlegroups.com.

[go-nuts] Re: How to work with multiple environment like dev, test, staging, prod in golang ?

[Ronny Bangsund]

On Sunday, May 17, 2020 at 7:19:23 PM UTC+2, Shishira Pradhan wrote:
>
> In springboot, we have configuration like port, database etc info are 
> stored in yaml files like application-dev.yml,   application-test.yml, 
> application-prod.yml profiles, and profile name is passed during running 
> the application to load profile specific configuration. I need to 
> understand that how to load/pass profile based properties in golang ?
>
I use Docker a lot at work (everything we do runs on AWS or Google 
servers), and that makes separating things relatively simple.

AWS has different environment variables for different configurations, 
allowing one Docker image to run with different settings. With 
docker-compose you can tie together tailored Dockerfiles and different 
images for different purposes, as needed. I recommend having a default 
"docker-compose.yml" be the dev environment, with its own environment 
variables, and variants for staging/prod with other names expecting envvars 
from the system it's running on. This makes the default invocations of 
docker-compose launch in local dev mode.

We use a lot of Python (Django, Wagtail) which expects envvars anyway, so 
it's a natural way to do things there. For Go I either write AWS-specific 
things which only expect envvars, or use a configuration system which 
overrides config files with outside vars.

We haven't started using Kubernetes yet, but the process probably won't be 
much different.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7c421559-8a25-450b-9e4f-b48e72f8f985%40googlegroups.com.

[go-nuts] How to dump cfg for method ?

[eric fang]

How to generate the cfg of the following Replace method ? I used this 
command "GOSSAFUNC=Replace:* go tool compile main.go" but it doesn't work. 
Thanks.

package main

type byteReplacer [256]byte

func (r *byteReplacer) Replace(s string) string {
var buf []byte // lazily allocated
for i := 0; i < len(s); i++ {
b := s[i]
if r[b] != b {
if buf == nil {
buf = []byte(s)
}
buf[i] = r[b]
}
}
if buf == nil {
return s
}
return string(buf)
}

func main() {
}

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5f599008-15aa-4a5f-b5fb-e9d899ffc315%40googlegroups.com.