Robert, Tracing is not possible, there will be trillions of instructions logged at the instruction level tracing.
Let me elaborate a bit more. The kind of tool I am thinking, intercepts every instruction being executed on all OS threads in the go process. The issue is that when an "interesting event" has been detected by my tool, I need to record the calling context, that is main()->func1()->func2(). The call stack unwinding is too expensive in my case since the "interesting events" happen too frequently. But it is much easier to maintain a running per-go-routime shadow stack by tracking call-ret sequences as it is executing on an OS thread. Since, the go scheduler switches from one go routine stack to another, I need to do the following things (in addition to the call-ret tracking): 1. Precisely recognize the stack switch event by the current thread. Let the current go stack be S1. 2. Save the shadow stack S1_shadow (analogous to the save sequence in the gosave routine) in metadata. 3. Precisely identify the other stack S2 the scheduler is switching to. 4. Search for S2_shadow in the metadata and switch to it (analogous to the restore sequence in the gogo routine). -Milind On Sat, Feb 9, 2019 at 5:39 PM robert engels <reng...@ix.netcom.com> wrote: > Milind, > > Understood. A few things to think about: 1) the tracing seems pretty > efficient in my tests, although if you don’t want a custom runtime, you’ll > need to post analyze on the file. 2) I think you are going to have a hard > time doing what you are trying to do because of the shared threads, you > won’t have an idea of what is being executed just by recording the place of > the context switch - you’ll need to inspect the associated ‘current g’ > structure, which can be different based on the compiled version of Go. > > It is a similar problem faced even by pprof. Since it only records ‘time > in function’ and isn’t reported by Go routine, just the combined process, a > Go routine that locks and OS thread and calls a system call that pauses, > looks on the outside as being expensive… when in reality it is not doing > anything. > > On Feb 9, 2019, at 7:07 PM, Milind Chabbi <milis...@gmail.com> wrote: > > Ian, > > Can you give me more details about these terms: "g0", "systemstack", and > "mcall"? > > Robert: taking the trace route is incorrect for me; I am profiling, > tracing is too expensive because of logging, furthermore, I don't want to > recompile the application or change the go runtime. > > -Milind > > On Sat, Feb 9, 2019 at 2:34 PM Ian Lance Taylor <i...@golang.org> wrote: > >> On Sat, Feb 9, 2019 at 2:02 PM <milis...@gmail.com> wrote: >> > >> > I am looking at fine-grained calling context collection for go lang >> programs (for all go routines) using binary instrumentation tools such as >> Intel Pin. >> > In a traditional language such as C/C++ intercepting CALL/RET >> instructions (plus some special handling for exceptions setjmp/longjmp) >> suffices. >> > Go makes it significantly more complicated. >> > >> > For Go, the scheduler can context switch from one goroutine to another >> (including garbage collection etc.). >> > The scheduler adjusts the stack pointer and program counter during >> these events, which (for x86) is mostly in this file: >> https://github.com/golang/go/blob/master/src/runtime/asm_amd64.s >> > Is there a go runtime expert, who can authoritatively confirm whether >> all the go routine context switching code is captured in this file or if >> there are other places too? >> >> Yes, for amd64 all the goroutine switching code is in runtime/asm_amd64.s. >> >> >> > It would also be great if somebody can confirm whether saving the >> current go routine state into gobuf_sp, gobuf_pc, gobuf_g, gobuf_ctxt, >> gobuf_ret and restoring a new one and jumping to the new gobuf_pc is the >> standard context switching idiom? Is there use of any other idiom such as >> overwriting the return address of a caller on the thread stack to jump to a >> different context during a return from a procedure? >> >> Yes, that is the standard idiom for switching goroutines, as seen in >> the gosave, gogo, and mcall functions. Also systemstack arguably >> changes goroutines, though only to the g0 of the current thread. >> >> The runtime does overwrite the PC in order to panic from a signal >> handler, in sigctxt.preparePanic, but not for goroutine switching. >> >> As Robert Engels says, tracking traceGoStart might be useful, though >> you do have to have tracing enabled. >> >> Ian (please reply to the mailing list, not just to me; thanks) >> > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
