[go-nuts] Re: Tooling experience feedback

[ondrej . kokes]

go tool trace was nicely explained here

https://groups.google.com/forum/#!topic/Golang-Nuts/Ktvua7AGdkI

and had no proper documentation at the time. Would be good if it had a high 
level overview like that

On Tuesday, 18 October 2016 19:54:49 UTC+1, Jaana Burcu Dogan wrote:
>
> Hello gophers,
>
> I am going to spend some time on improving the not-documented or 
> hard-to-use parts of the Go tools. I will mainly focus on go build and go 
> test. The work will consist of reorganizing manuals, proposing new 
> subcommands or flags to support commonly used multi-step things and so on.
>
> To give you some concrete examples:
>
> I have been in cases where users cannot find the build flags because they 
> are hidden. E.g.
>
> $ go test -help
> test [build/test flags] [packages] [build/test flags & test binary 
> flags]
> ...
>
> requires you to know about where the build flags are. Some tools are 
> invisible for the newcomers.
>
> The other example is about test coverage. It is so tedious that I need an 
> alias.
>
> func gocover() {
> go test -v -coverprofile=coverage.out && go tool cover 
> -html=coverage.out
> } 
>
> I want to learn more about your experience, aliases, etc to propose 
> improvements and work on the most commonly suggested items. I also am 
> planning to go through this mailing list, stack overflow and other channels 
> to see the common complaints.
>
> Feel free to reply to this thread or write to me privately.
>
> Thanks,
> JBD
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Generating execution trace diagram for a go program

[ondrej . kokes]

This is excellent, really helpful. The -help flag only mentions testing and 
casual searching only again pops up with info on more testing, I couldn't 
see anything on actually using it for running a generic binary. Perhaps it 
would be worth adding this to the documentation?

+ things like not needing the binary as an argument since 1.7 etc.

Or maybe just an example in the package 
file https://golang.org/pkg/runtime/trace/

On Thursday, 20 August 2015 11:40:49 UTC+1, Egon wrote:
>
> On Wednesday, 19 August 2015 23:32:16 UTC+3, Shan Valleru wrote:
>>
>> In Go 1.5, it seems like you can generate a execution flow diagram (like 
>> this - https://talks.golang.org/2015/dynamic-tools/trace.png) for a go 
>> program using trace command .
>> Anybody know the detailed steps for generating such a flow diagram from a 
>> go program?
>>
>
> Here's the code I used to create a 10s trace from a server. You should be 
> able to modify it to your own needs.
>
> import "runtime/trace"
>
> func trace10(w http.ResponseWriter, r *http.Request) {
> f, err := os.Create(time.Now().Format("2006-01-02T150405.pprof"))
> if err != nil {
> panic(err)
> }
> defer f.Close()
>
> if err := trace.Start(f); err != nil {
> panic(err)
> }
> defer trace.Stop()
>
> // All the important stuff is happening in other goroutines so we just 
> wait here
> time.Sleep(10 * time.Second)
> }
>
> Once you have integrated that into your own code whatever way you need:
>
> // on Linux you can create/analyze it as:
> go build .
> ./program
> go tool trace program 2015-08-20T133508.pprof
>
> // on Windows
> go build .
> program.exe
> go tool trace program.exe 2015-08-20T133508.pprof
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Data locality in large slices

[ondrej . kokes]

(I have now recreated it on my Mac, under 1.7rc5, the runtime differences 
are still there.) I thought the compiler was removing these as you suggest, 
but then StartEnd and EndStart had wildly different running times, despite 
using the very same values. So I added dummy assignments to double check 
and nothing has changed

https://play.golang.org/p/kpsyWHkFBE

So it seems the behaviour changed between 1.6 and 1.7 - either it better 
hints the CPU, or it eliminates some instructions in the new SSA backend.

On Wednesday, 3 August 2016 23:15:05 UTC+1, Dave Cheney wrote:
>
> You need to use the values to ensure that compiler does not remove the 
> code. Even if the compiler does not do this, your Intel CPU will, if 
> effectively runs an ssa implementation in hardware and will spot the dead 
> stores and skip the load.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Data locality in large slices

[ondrej . kokes]

Downgrading to 1.6.3, I'm also getting consistent benchmark results. I'll 
try 1.7 on my Mac at home later today, to see if it's a 1.7 thing or a 
Windows thing or...?

On Wednesday, 3 August 2016 14:55:20 UTC+1, C Banning wrote:
>
> PS - that's with Go v1.6.
>
> On Wednesday, August 3, 2016 at 7:49:49 AM UTC-6, C Banning wrote:
>>
>> On MacBook Pro, 2.6 GHz Intel Core i7, 8 GB 1600 MHz memory, running OS X 
>> 10.11.6, your benchmarks look pretty consistent:
>>
>>
>> BenchmarkStart-4  20 1.45 ns/op
>>
>> BenchmarkEnd-420 1.47 ns/op
>>
>> BenchmarkHereThere-4  20 1.46 ns/op
>>
>> BenchmarkStartEnd-4   20 1.46 ns/op
>>
>> BenchmarkEndStart-4   20 1.46 ns/op
>>
>> BenchmarkFirst-4  20 0.59 ns/op
>>
>> BenchmarkSecond-4 20 0.59 ns/op
>>
>> BenchmarkLast-4   20 0.59 ns/op
>>
>> BenchmarkPenultimate-4 20 0.58 ns/op
>>
>> On Wednesday, August 3, 2016 at 5:56:32 AM UTC-6, Ondrej wrote:
>>>
>>> I wanted to see if there was a difference when loading values from a 
>>> large-ish slice (1 elements) - to see if caches, locality and other 
>>> things had any meaningful impacts. Whilst individual value loading (just a 
>>> single element) seemed to be equally fast regardless of element position 
>>> (see bench of First, Second, Last, Penultimate), when combining loading of 
>>> various values, there seem to be almost a 2.5x difference between loading 
>>> first four values and loading last four values (first two benchmarks).
>>> Loading the same values, just in different order, also yields different 
>>> execution times. But alternating loading (0, n, 1, n-1) seems to be faster 
>>> than loading first two values and last two values.
>>>
>>> (Setting the test slice to be an array instead wipes all differences 
>>> between benchmarks.)
>>>
>>> Can anyone point me to a resource - be it Go specific or on computer 
>>> science principles - that would explain these large differences?
>>>
>>> Thanks!
>>>
>>> https://play.golang.org/p/oMqDvXI9YW
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: An efficient runtime expression evaluation

[ondrej . kokes]

I think this pretty much covers it

_, err := VirtualProtect(fn.body, 0x40)
if err != nil {
panic(err)
}

// OH GOD WHAT HAVE I DONE???
type callstub struct{ fn func(*Memory) }

Excellent and worrying work :-) A 2x difference between interpreted and 
native is pretty rad. It strongly reminds me of the JIT endeavours I 
researched earlier today - none of those seemed to have survived. I'd wait 
until there's native support before pushing things like these into 
production. But it would be nice to reopen the discussion about runtime 
(portable) assembly, could make performance critical code in the 
text/numerical analysis sphere quite a bit easier.

As for all the other developments in this thread - I'll go through them one 
by one - there are subtle things (operators from int to byte, which makes 
for faster comparisons; caching of value lookups) and less subtle things 
(VM), where I'll have to consider the ups and downs. After all, my example 
is quite cut down and the actual production version has a lot more 
complexity bolted on with a lot more data and fast turnaround (so caching 
and small VMs might not be worthwhile). In any case, lot of excellent 
advice all around, thank you, sir.

On Monday, 18 July 2016 17:07:27 UTC+1, Egon wrote:
>
> On Monday, 18 July 2016 13:27:09 UTC+3, Michael Jones wrote:
>>
>> Anything much faster than this needs vector operations in the interpreter 
>> so the “get to the OP function” overhead is once per time series rather 
>> than once per element in the series.
>>
>
> I managed to make it a little faster https://play.golang.org/p/Tr2PRKI23w, 
> but runs only on windows (kind of).
>
> *What have I done...*
>
> + Egon
>
>  
>
>>  
>>
>> *From: * on behalf of Egon 
>> *Date: *Monday, July 18, 2016 at 1:32 AM
>> *To: *golang-nuts 
>> *Cc: *
>> *Subject: *[go-nuts] Re: An efficient runtime expression evaluation
>>
>>  
>>
>>
>>
>> On Monday, 18 July 2016 11:13:08 UTC+3, Egon wrote:
>>
>>
>>
>> On Monday, 18 July 2016 10:30:14 UTC+3, Egon wrote:
>>
>> On Monday, 18 July 2016 03:11:29 UTC+3, ondrej...@gmail.com wrote:
>>
>> Cheers, I tried replicating my endeavours (
>> https://play.golang.org/p/Qxoo2ASac6), sorry if it's still too verbose. 
>> It's essentially rewriting the inbuilt ast.Node into a simpler nested 
>> struct and then walking it.
>>
>>  
>>
>> In testing the performance, I started adding algebraic expressions, which 
>> make my walking more expensive, but don't change the 'native' expression 
>> evaluation (I guess due to constant folding).
>>
>>  
>>
>> As to your suggestion three - I do the variable lookup in the parsing 
>> stage, but I still need to retain the pointer, not the value itself, 
>> because I'm accessing an element of that given variable (time series), and 
>> this element (time period) changes at runtime.
>>
>>  
>>
>> https://play.golang.org/p/dd4hTpMKrp
>>
>>  
>>
>> Of course you can additionally add constant folding and similar... 
>> Additionally instead of working on a single float at a time, make each 
>> variable an array of 8 floats, that are computed in parallel.
>>
>>  
>>
>>  
>>
>> Just realized that it's trivial to write basic constant folding: 
>> https://play.golang.org/p/iqWX5_Mweb
>>
>>  
>>
>> Although it probably will be easier to maintain and extend as a separate 
>> pass: https://play.golang.org/p/xcz5mXoaOG
>>
>>  
>>
>>  
>>
>> This brings the result to:
>>
>>  
>>
>> interpreter: 17.001ms
>>
>> native: 7.0004ms
>>
>>  
>>
>> Which is approximately the best I would expect from an interpreter 
>> without JIT (and not computing multiple time-points at a time).
>>
>>  
>>
>> + Egon
>>
>>  
>>
>> One performance gain I can think of is to implement some pruning through 
>> the abovementioned constant folding and other optimisations, but I'd rather 
>> leave that as the last resort. Another thing that comes to mind is that I 
>> could return nested closures in some way - meaning that '1+3*x' would be, 
>> in go-like pseudocode, add(func() { return one }, func mul(func() { return 
>> three}, func() {return model[x]} )), where the one/tree are values passed 
>> to the closure when parsing the equation; but that's just now off the top 
>> of my head.
>>
>>  
>>
>> I attached a pprof result in the header.
>>
>>  
>>
>> Thanks again.
>>
>>
>> On Friday, 8 July 2016 15:46:32 UTC+1, Egon wrote:
>>
>> On Friday, 8 July 2016 16:25:40 UTC+3, Ondrej wrote:
>>
>> Hi all,
>>
>> I have a model with variables, let's call them a, b, c, ..., z. These are 
>> numerical values (time series loaded from a database) and I let the user 
>> specify their relationships in a JSON, say 'z = 12; x = a + 2/3 + 3*c; y = 
>> log(12*f) + exp(g)' etc. The syntax is trivial - it's basically just 
>> algebraic relationships + a few functions (log, log2, log10, exp, 
>> trigonometrics, ...; all 1:1 mappings to their math package equivalents).
>>
>>  
>>
>> *Tip: 

[go-nuts] Re: An efficient runtime expression evaluation

[ondrej . kokes]

Cheers, I tried replicating my endeavours 
(https://play.golang.org/p/Qxoo2ASac6), sorry if it's still too verbose. 
It's essentially rewriting the inbuilt ast.Node into a simpler nested 
struct and then walking it.

In testing the performance, I started adding algebraic expressions, which 
make my walking more expensive, but don't change the 'native' expression 
evaluation (I guess due to constant folding).

As to your suggestion three - I do the variable lookup in the parsing 
stage, but I still need to retain the pointer, not the value itself, 
because I'm accessing an element of that given variable (time series), and 
this element (time period) changes at runtime.

One performance gain I can think of is to implement some pruning through 
the abovementioned constant folding and other optimisations, but I'd rather 
leave that as the last resort. Another thing that comes to mind is that I 
could return nested closures in some way - meaning that '1+3*x' would be, 
in go-like pseudocode, add(func() { return one }, func mul(func() { return 
three}, func() {return model[x]} )), where the one/tree are values passed 
to the closure when parsing the equation; but that's just now off the top 
of my head.

I attached a pprof result in the header.

Thanks again.

On Friday, 8 July 2016 15:46:32 UTC+1, Egon wrote:
>
> On Friday, 8 July 2016 16:25:40 UTC+3, Ondrej wrote:
>>
>> Hi all,
>> I have a model with variables, let's call them a, b, c, ..., z. These are 
>> numerical values (time series loaded from a database) and I let the user 
>> specify their relationships in a JSON, say 'z = 12; x = a + 2/3 + 3*c; y = 
>> log(12*f) + exp(g)' etc. The syntax is trivial - it's basically just 
>> algebraic relationships + a few functions (log, log2, log10, exp, 
>> trigonometrics, ...; all 1:1 mappings to their math package equivalents).
>>
>
> *Tip: include a working piece of code that you want to make faster, it 
> makes it easier for people to see the problems and common issues.*
>
>
>> Now, I get these relationships in a JSON and I parse them using 
>> go/parser. Then I walk the tree once and process it a bit - replacing 
>> keywords by pointers to my variable stores, replacing all the log/exp/sin 
>> with function pointers, leaving literals be literals etc. Each node is then 
>> a struct with a type and the actual contents (sadly a generic interface, 
>> because the value can be almost anything). The prep stage is now over.
>>
>> When actually running the model, I loop through years and within each 
>> year I solve each variable - I walk the tree and evaluate it where needed. 
>> The only non-trivial action is when I get to a model variable, I need to do 
>> a bit of lookup (it's a time series, so I need to look up the correct time 
>> period and other bits). Otherwise it's just literals, operators and 
>> function calls, all of which is fairly straightforward.
>>
>> This is all well and good. One of the issues is that it's rather slow. I 
>> thought it would be the recursive nature (and interface assertions), but 
>> converting all this into a shunting yard system didn't improve the 
>> performance dramatically. I've profiled the thing and removed a few 
>> hotspots, my question is not about profiling. I'm after a bit more general 
>> advice on how to handle these runtime evaluations and if there are better 
>> ways of doing so. Essentially some sort of a JIT (but Go does not have 
>> runtime assembly, right?), or maybe convert each expression into a closure 
>> or maybe a whole different algorithm or...?
>>
>
> Reduce the amount of code and indirection that you need to do, few basic 
> ideas:
> 1. implement a VM https://play.golang.org/p/dlmZ2lGPY7
> 2. operate on vectors of variables instead of single values 
> https://play.golang.org/p/25MIjIXs0D
> 3. try to do the lookup of all necessary variables before starting to 
> compute with them; if possible
>
> Obviously pprof is your friend. (
> https://blog.golang.org/profiling-go-programs)
>
> + Egon
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.