yes On Tue, Mar 24, 2020 at 11:48 AM Orson Cart <objectivedynam...@gmail.com> wrote:
> On Tuesday, 24 March 2020 18:31:29 UTC, Michael Jones wrote: >> >> You use them to stop the time charged against your benchmark. >> >> For example: >> >> bench: >> stop timer >> generate initial data >> start timer >> do test >> > > Thanks Michael. What if the initial data has to be generated from scratch > on each iteration? Is the following expected to work? > > bench: > b.N Loop > stop timer > generate new data > start timer > do test > > > > >> >> On Tue, Mar 24, 2020 at 10:47 AM Orson Cart <objectiv...@gmail.com> >> wrote: >> >>> >>> On Tuesday, 24 March 2020 16:49:55 UTC, Robert Engels wrote: >>>> >>>> Can you please succinctly explain the problem? >>>> >>> >>> Let's see. Benchmarks can yield incorrect results when b.StopTimer and >>> b.StartTimer are used. >>> >>> My reasoning: >>> >>> 1/ I have a benchmark that calls a single function. The reported >>> duration is T. >>> 2/ I have another benchmark which invokes the same function twice. I'd >>> expect the reported duration for this to be nominally 2T and within reason >>> it appears to be the case. >>> 3/ I have yet another benchmark which also invokes the same function >>> twice but it stops the benchmark timer before the first call and then >>> starts it after the the first call has completed. I'd expect the reported >>> duration to be nominally T again. It isn't. It's closer to 3T on my >>> hardware and I've seen much worse correlation on other hardware - almost 5T >>> >>> Now it's entirely possible that I'm misunderstanding something about how >>> b.StopTimer and b.StartTimer are intended to be used, hence my post here :) >>> >>> >>> >>> >>> >>> >>> >>>> >>>> >>>> On Mar 24, 2020, at 11:24 AM, Orson Cart <objectiv...@gmail.com> wrote: >>>> >>>> >>>> I posted this earlier but I realised that the code had a fundamental >>>> error in it. I've corrected here it but the underlying problem still >>>> exists. >>>> >>>> I've recently started using go test's benchmarks support and I'm >>>> particularly interested in understanding the benchmark timer functions. >>>> I've been getting results that I found surprising and I was wondering if >>>> anyone could explain what's going on here. >>>> >>>> The code below has three benchmarks that each invoke a single function >>>> (foo). The implementation of foo isn't important, it's just there to >>>> consume some time: >>>> - foo is called once per iteration in Benchmark1. >>>> - It's called twice per iteration in Benchmark2 so I'd expect >>>> Benchmark2's duration to be nominally twice that of Benchmark1. >>>> - It's also called twice per iteration in Benchmark3 but the first call >>>> is wrapped in b.StopTimer and b.startTimer calls. Because of this I'd have >>>> expected Benchmark3 to be about the same duration as Benchmark1 >>>> >>>> Apologies for the length of the example but I didn't think it fair to >>>> ask the question and leave anything out. >>>> >>>> package demo_test >>>> >>>> import ( >>>> "strconv" >>>> "testing" >>>> ) >>>> >>>> var Foo1 []string >>>> var Foo2 []string >>>> var Count int = 32767 >>>> >>>> func Benchmark1(b *testing.B) { >>>> for i := 0; i < b.N; i++{ >>>> Foo1 = foo(Count) >>>> } >>>> } >>>> >>>> func Benchmark2(b *testing.B) { >>>> for i := 0; i < b.N; i++{ >>>> Foo1 = foo(Count) >>>> Foo2 = foo(Count) >>>> } >>>> } >>>> >>>> func Benchmark3(b *testing.B) { >>>> for i := 0; i < b.N; i++{ >>>> b.StopTimer() >>>> Foo1 = foo(Count) >>>> b.StartTimer() >>>> Foo2 = foo(Count) >>>> } >>>> } >>>> >>>> func foo(count int) []string{ >>>> testData := []string{} >>>> for i:= 0; i < count; i++ { >>>> testData = append(testData, strconv.Itoa(i)) >>>> } >>>> >>>> return testData >>>> } >>>> >>>> >>>> When the benchmarks are run the results are as follows: >>>> >>>> Benchmark1-4 351 3345215 ns/op >>>> Benchmark2-4 166 7206582 ns/op >>>> Benchmark3-4 334 3457907 ns/op >>>> PASS >>>> ok bar.com/benchmarks 6.881s >>>> >>>> OK benchmark3 is a little slower than Benchmark1 but that's not what's >>>> bothering me. It's this: if I now change Count to something much smaller >>>> the results are a surprise, at least to me. Here are the results when Count >>>> = 8: >>>> >>>> Benchmark1-4 2706196 442 ns/op >>>> Benchmark2-4 1357482 873 ns/op >>>> Benchmark3-4 840729 1387 ns/op >>>> PASS >>>> ok bar.com/benchmarks 23.547s >>>> >>>> The ratio of timings for Benchmark1 and Benchmark2 are roughly in line >>>> with expectations but I was surprised to see that the timings for >>>> Benchmark3 are now larger than those for Benchmark2. >>>> >>>> Can anyone explain this? >>>> >>>> TIA >>>> Orson >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golan...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/38f10940-a770-4fc5-86a9-19cc1371152a%40googlegroups.com >>>> <https://groups.google.com/d/msgid/golang-nuts/38f10940-a770-4fc5-86a9-19cc1371152a%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golan...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/1d291cef-420a-4f80-b719-e0654585c25d%40googlegroups.com >>> <https://groups.google.com/d/msgid/golang-nuts/1d291cef-420a-4f80-b719-e0654585c25d%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> >> *Michael T. jonesmichae...@gmail.com* >> > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/c3fcc6c5-9aa2-489c-adc4-c6b070ea6617%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/c3fcc6c5-9aa2-489c-adc4-c6b070ea6617%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>* -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CALoEmQyLgHddM9p%3DkJO1R-LojUq2DwPAMx%2BS94Dt4_uR1ypVTA%40mail.gmail.com.