Hi. I was one of the people who failed in an attempt to auto-insert contexts in all of google3. I no longer remember all the obstacles I failed to overcome, but would encourage others to take on the project.
One issue was libraries that were used both in paths from http requests (so they needed the context propagated through them) but were also used in places where their callers (starting at main) didn't have any contexts at all. Sitting by my coffee this morning this no longer seems like an insuperable obstacle (e.g., pass nils and change them to context.Background() inside, or, like appengine's log, split packages into one that requires contexts and one that doesn't). But I had lots of what I thought were plausible ideas that broke on the hard rock of google3, and I suspect these would too. (The project wasn't a total failure. It did plumb contexts through a bunch of packages. But the general case was too hard for me.) On Sat, May 13, 2017 at 3:50 AM, <mhhc...@gmail.com> wrote: > A reference i found about "reactive programming" > https://github.com/dsyer/reactive-notes/blob/master/intro.adoc > > Hope this introduced the concept correctly, thanks for pointing that. > > Following are only some thoughts, > > Two things surprising here, > 1/ a programmer prefers not to solve a problem > 2/ one failed attempt mean the end of all attempts** > > That being said, my little understanding so far is that current context is > used for two purposes, > - cancellation > - data (transmission? not sure) (+/- ts, have not checked i expect so in > regards to previous referenceS cited) > > While cancellation is a feature i really want to take advantage of, > the other one is much more arguable in the sense that > it can remains expressed purely by the end user without bloating > his productivity and quality (you don t really need it everywhere, just > where it is needed), > it seems to be highly software context dependent. > > Whereas cancellation, while it looks likes simple, is maybe more subtle. > After all it is about assigning uuid to a chain of call and > appropriately propagate, disjoint/rejoin new uuids with the previous one, > so that we can ask to stop execution of a sub selection of a chain of calls > via an handle. > Am i too simplistic? > > Its difficulty reside in its requirement to be passed absolutely > everywhere, > That reveals an important fact about the nature of cancellation, > it is there, it is everywhere, at all time (...within the program > lifetime), > it is somewhere in the background of every ops since the very beginning > the program began, > but not necessarily enabled, and certainly not a straight line. > > That is where the syntax might help to establish the plots > that the runtime consumes to connect the dots > and support what the developers want to achieve, > in my understanding so far. > > My best comparison to cancellation is > request rays tracing in micro services oriented architecture, > on both end it is a multi-tenant, > it always start with one ray, > the ray always split into multiple rays, > because we do programming, we need ways > to create, control distribute existing/new rays, > and possibly let the userland introduce a new behavior for the rays. > > So yeah basically main has a ray, > if you process an http request, > you to create a new ray > to be able to cancel only that request, > but it needs to be connected to the main ray > because if main should be canceled, > that signals should propagate to all rays connected with it, > probably. > > ** i want to put emphasis because in the description provided > by Sameer, as i tried to summarize, It has been tried to handle > the problem via the type system as if it was > a completely defined set of types. > Which in go is wrong (yeah i m very affirmative here :), > from my understanding, that might be correct in other oop languages. > Technically it is possible to rewrite interfaces, methods signatures, > taken individually, > as a whole, and from the consumer pov, > i d say it is an impossible task in go because it it goes against its > nature. > And i confirm/understand that by reading to Sameer feedback. > > Notes: i m a strong lover of go type system (not talking about values and > pointers ;) > > > On Friday, May 12, 2017 at 4:42:54 PM UTC+2, Henrik Johansson wrote: >> >> With the whole "Reactive" movement thread locals have started to vanish >> at least in the Java ecosystem. >> I agree with Sameer that, while convenient, it comes with a whole set of >> obscure bugs. >> >> On Fri, May 12, 2017, 14:57 Sameer Ajmani <sam...@golang.org> wrote: >> >>> Hmm, I'm not finding good open-source examples of ThreadLocal context >>> propagation in C++ and Java. My experience with this is based on what >>> Google uses internally. Perhaps someone more familiar with context use >>> (tracing?) outside Google can chime in? +Jaana Burcu Dogan >>> >>> On Thu, May 11, 2017 at 7:02 AM <mhh...@gmail.com> wrote: >>> >>>> thanks, >>>> >>>> ..integrating Context into the runtime.. >>>> >>>> >>>> 50% runtime, 50% syntax with explicit contextualization. >>>> >>>> ..The flow of request processing in Go may include multiple goroutines >>>> and may move across channels; >>>> >>>> >>>> yes big ? mark here. might the 50% of an handy and explicit syntax help >>>> with it? >>>> >>>> >>>> C++ and Java use thread-local Contexts, and while this is convenient, >>>> it is often a source of mysterious bugs. >>>> >>>> thanks! I don't know them, >>>> >>>> I quickly checked according to this >>>> https://dzone.com/articles/painless-introduction-javas-threa >>>> dlocal-storage >>>> I may have totally wrong, the syntax does not look like, not even from >>>> a 100km, >>>> to how i represent go contexts in my head. >>>> >>>> This is more like the actor pattern dave cheney talks about in >>>> https://dave.cheney.net/2016/11/13/do-not-fear-first-class-functions >>>> (search for Let’s talk about actors) >>>> >>>> Is the dzone link correctly describe >>>> what you mentioned as being go context equivalent in java ? >>>> >>>> sorry my questions are so basic. >>>> >>>> >>>> On Thursday, May 11, 2017 at 12:29:11 PM UTC+2, Sameer Ajmani wrote: >>>> >>>>> I think you are asking whether we considered integrating Context into >>>>> the runtime, so that it does not need to be passed explicitly. Yes, we >>>>> discussed this, but decided against it. The flow of request processing in >>>>> Go may include multiple goroutines and may move across channels; we >>>>> decided >>>>> an explicit Context made this much easier to get right. C++ and Java use >>>>> thread-local Contexts, and while this is convenient, it is often a source >>>>> of mysterious bugs. >>>>> >>>> On Thu, May 11, 2017 at 4:08 AM <mhh...@gmail.com> wrote: >>>>> >>>>>> Thanks a lot! >>>>>> >>>>>> Might i guess and try to generalize your explanations into >>>>>> "we tried to write a plumber for all cases possible" >>>>>> >>>>>> Which matters a lot, in my humble opinion. >>>>>> >>>>>> At least for the various reasons you put there, >>>>>> simply put, >>>>>> because it seems not technically achievable. >>>>>> >>>>>> Still i m happy you gave me those details, they are of interest >>>>>> indeed. >>>>>> >>>>>> I rather intent to solve the problem on a smaller surface >>>>>> with more predictability, less impossible solution to find. >>>>>> I hope so. >>>>>> And if it saves 90% of the time, that s already a win, imho. >>>>>> >>>>>> May i ask another question, >>>>>> have you considered to create the plumbing >>>>>> at runtime rather than statically ? >>>>>> With some intents from the syntax, necessarily >>>>>> (and yeah go 1 compat will be a problem, let s forget it for 1 >>>>>> minute). >>>>>> >>>>>> I suspect in some aspects in might be handier, >>>>>> because it might be all about chained func calls and not type system >>>>>> handling, >>>>>> and it might be easier to interleave the ctx.check in the flaw of ops, >>>>>> I don t know enough to realize for sure. >>>>>> >>>>>> >>>>>> >>>>>> On Wednesday, May 10, 2017 at 11:40:27 PM UTC+2, Sameer Ajmani wrote: >>>>>> >>>>>>> Our approach was to identify function calls that consume a Context >>>>>>> (indicated by a call to context.TODO in the function body) and function >>>>>>> calls that provide a Context (such as RPC and HTTP handlers). Then we >>>>>>> use >>>>>>> the guru callgraph tool to find all call paths from context providers to >>>>>>> context consumers. These are the call paths that need to have a context >>>>>>> plumbed through them. >>>>>>> >>>>>>> Starting from a context consumer, we can work upward to the context >>>>>>> provider, adding a context parameter to reach intervening function >>>>>>> signature. Replace context.TODO in the consumer function body with the >>>>>>> new >>>>>>> ctx parameter, then update all the places that call the consumer to pass >>>>>>> context.TODO. Now we have a new set of context consumers. Repeat until >>>>>>> you >>>>>>> reach the context providers (if you reach main or a Test function, pass >>>>>>> context.Background instead). >>>>>>> >>>>>>> This works OK for static function calls but gets messy for dynamic >>>>>>> calls. If you need to add a context parameter to an interface method, >>>>>>> now >>>>>>> you have to update all implementations of that method, too (guru can >>>>>>> find >>>>>>> these for you). And if that interface is outside your control (like >>>>>>> io.Writer), you cannot change its signature, so you have to pass the >>>>>>> context some other way (such as via the method receiver). >>>>>>> >>>>>>> This gets yet more complicated if you cannot make atomic changes to >>>>>>> all callers of your functions, because callers may be in other >>>>>>> repositories. In this case, you must do an incremental refactoring in >>>>>>> multiple steps: each change to a function signature involves adding a >>>>>>> new >>>>>>> function that has the context parameter, then changing all existing >>>>>>> calls >>>>>>> to use the new function, while preventing new calls to the old >>>>>>> function, so >>>>>>> that you can finally delete it. >>>>>>> >>>>>>> Inside Google, we ended up not needing to build all this: Context >>>>>>> was introduced early enough that Go users could plumb it manually where >>>>>>> needed. I think a context plumbing tool could still be interesting and >>>>>>> useful to other Go users. I'd love to see someone build it! >>>>>>> >>>>>>> S >>>>>>> >>>>>>> On Tue, May 9, 2017 at 10:54 AM <mhh...@gmail.com> wrote: >>>>>>> >>>>>>>> > I've done a limited form of this using awk ;-) >>>>>>>> >>>>>>>> if you have a minute, >>>>>>>> >>>>>>>> can you tell more about what limited you >>>>>>>> in your attempts and which trade made you stop (guessing), >>>>>>>> if any ? >>>>>>>> >>>>>>>> Do you still think it be awesome ? >>>>>>>> Or have you made your mind to an opposite position ? >>>>>>>> if so, For which reasons? >>>>>>>> >>>>>>>> My tool is very poor, consider it as on going, a place for >>>>>>>> inspiration to get started from absolutely no idea to lets get a dirty >>>>>>>> prototype. >>>>>>>> not sure yet how long is going to be the road, still digging :) >>>>>>>> >>>>>>>> >>>>>>>> On Tuesday, May 9, 2017 at 4:25:46 PM UTC+2, Sameer Ajmani wrote: >>>>>>>> >>>>>>>>> The eg tool can execute simple refactoring steps, but automating >>>>>>>>> context plumbing through a chain of calls is an open problem. Alan >>>>>>>>> Donovan >>>>>>>>> put some thought into this a few years ago, and I've done a limited >>>>>>>>> form of >>>>>>>>> this using awk ;-) >>>>>>>>> >>>>>>>> On Tue, May 9, 2017 at 6:10 AM <mhh...@gmail.com> wrote: >>>>>>>>> >>>>>>>> I want something similar too. >>>>>>>>>> >>>>>>>>>> Automatic and smart insertion of context args in a chain of calls. >>>>>>>>>> >>>>>>>>>> Methods signature updates are easy, but how to appropriately >>>>>>>>>> insert context check in the ast ? >>>>>>>>>> I m not sure yet. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >The difficulty here seems to differentiate intra package calls >>>>>>>>>> from calls to standard/3rd party libraries which shouldn't be having >>>>>>>>>> new >>>>>>>>>> param. >>>>>>>>>> >>>>>>>>>> That does not sound too difficult, from the pkg identifier, >>>>>>>>>> lookup for the import path, for every import path, exists in GOROOT ? >>>>>>>>>> >>>>>>>>>> Please put updates here anything you want to share. >>>>>>>>>> >>>>>>>>>> At that moment i m using this package to help me with ast, >>>>>>>>>> https://github.com/mh-cbon/astutil >>>>>>>>>> >>>>>>>>>> might be a start even though it needs refactoring. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Monday, May 8, 2017 at 1:03:52 AM UTC+2, meir fischer wrote: >>>>>>>>>>> >>>>>>>>>>> I'm adding tracing to an existing code base with many packages >>>>>>>>>>> and it seems the best way to have context's passed around is to >>>>>>>>>>> just have >>>>>>>>>>> every method take a context.Context. >>>>>>>>>>> >>>>>>>>>>> Is there any tooling for converting a code base/package to have: >>>>>>>>>>> (a) context.Context as the first parameter in each function - >>>>>>>>>>> ctx context.Context >>>>>>>>>>> (b) for any function that has changed, have its callers (within >>>>>>>>>>> that package) pass ctx as the first arg >>>>>>>>>>> >>>>>>>>>>> The difficulty here seems to differentiate intra package calls >>>>>>>>>>> from calls to standard/3rd party libraries which shouldn't be >>>>>>>>>>> having new >>>>>>>>>>> param. >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "golang-nuts" group. >>>>>>>>>> >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>>>> >>>>>>>>> >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "golang-nuts" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golang-nuts...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
