Ah! I think I have the kernel of a usable idea here. Thanks. How does this sound: template 1 outputs template 2. Input to template 1 kicks off the goroutines and places {{ get_results token_xyz }} as the output, but also has an output channel that we can wait on for all of the answers Then, template 2 is executed with a function that pulls the output from the channel. Sort of hard to explain, but seems on the face to be workable. I'll post results after I prototype it.
-- Michael On Wednesday, May 31, 2017 at 1:23:40 PM UTC-5, Egon wrote: > > On Wednesday, 31 May 2017 20:34:56 UTC+3, Michael Brown wrote: >> >> This is almost certainly going to be a case where you have the correct >> solution and I don't have the go experience to properly understand it. I >> wasn't quite understanding how your code was "patching" the results and how >> the template package knows to wait for it. >> > > Let's say you have a template > > {{ sleep }} xxx {{ sleep }} > > Once you render with that approach you get something like this... > > <<token1>> xxx <<token2>> > > So instead of returning the actual value you replace them with a token. > And spawn a go routine to fetch the answers. > > You wait for the results or just some, doesn't matter. > > Once you get a results, you replace it in the buffer... > > token1 -> "alpha" > token2 -> "beta" > > "alpha" xxx "beta" > > Or whatever the result is. > > However this approach is pretty limited when you want to range over the > results. > > Doing two passes over the template, might work better, but needs somewhat > better logic to handle lookup; but that could work for handling multiple > results... > > {{ sleep }} > > In the first pass you start the goroutines and each output goes into an > array of channels > > var rpcs = []func() interface{} > > var funcMap = template.FuncMap { "sleep": func() { rpcs = append(rpcs, > sleep) }, } > > // now you schedule goroutines in whatever way you need for the rpcs and > write the results to a channel > var results = []chan interface{} > > the second pass would do it as > current := 0 > > var funcMap = template.FuncMap { "sleep": func() interface{} { > > result := <-results[current] > > current++ > > return result > > > // ... > > > Without knowing the actual template, it is hard to recommend something > better simpler or easier. > > Let me describe my problem. >> >> I have a REST interface running on an embedded system. The REST api is >> mandated by another entity and I have zero control over the output, I have >> to produce specific conformant output. The data that I need to build the >> interface is on the system in a variety of other processes, and I can get >> that data via a couple of different RPC mechanisms, however sometimes these >> RPC mechanisms can be slow. (I'm in progress building go bindings for them). >> >> The current code which creates the REST interface is a huge morass of C >> code that generates the JSON output, so the exact structure of the JSON >> output is hardcoded in the code. >> >> I have an opportunity here, and that is that it appears to me that, with >> a little work, I can completely templatize the rest output. That is, I can >> have zero page-specific code and render almost all of the output by >> providing a small number of generic data access functions. The issue I ran >> into is the serial nature of the substitution kills the performance in my >> prototype. >> >> Yes, I can make a specific go function per page that gathers all the data >> I need and provides it to the template. But that would mean that I'd need >> to maintain both the templates and the data access functions. This is still >> an improvement on the old way of doing things, but I was hoping to jump >> straight to a fully templatized system, which according to my initial >> analysis, would be considerably less code and would not really be abusing >> the template system (most of the function calls are very straightforward >> and there is no heavy processing of the output). >> >> -- >> Michael >> >> On Wednesday, May 31, 2017 at 10:56:57 AM UTC-5, Egon wrote: >>> >>> Both of my described approaches run the funcs serially however it does >>> not wait for the response and later patxhes the results. >>> >>> Can you describe the whole thing you are building? Piecing the >>> requirements and purpose together from comments is difficult. >>> >>> I.e. how much memory, how big is request latency, who gets the output, >>> where do the templates come from... >>> >>> On May 31, 2017 6:40 PM, "Michael Brown" <michael...@gmail.com> wrote: >>> >>> The best thing I can think of is to modify text/template to add futures >>> support, and then do multi-pass rendering. The place to add this looks >>> relatively simple, however the implementation looks complicated (and I just >>> wrote my first program in go a couple weeks ago...) >>> >>> The problem that I see with the solution below is that text/template >>> execution tries to instantiate the values of each function serially and >>> essentially waits for completion on each, serially. I've spent the last >>> couple hours examining the text/template implementation. >>> -- >>> Michael >>> >>> >>> On Wednesday, May 31, 2017 at 8:59:20 AM UTC-5, Egon wrote: >>>> >>>> The best idea I can think of (without digging and modifying >>>> text/template) is to use special tokens and replace them afterwards... >>>> Of course this approach has limitations in what it can do. >>>> >>>> *// note: code untested and incomplete* >>>> >>>> type Token string >>>> type Queries struct { >>>> pending sync.WaitGroup >>>> mu sync.Mutex >>>> responses map[Token]string >>>> } >>>> >>>> func (q *Queries) createToken() Token { >>>> return unique token >>>> } >>>> >>>> func (q *Queries) Do(fn func() string) Token { >>>> token := q.createToken() >>>> q.pending.Add(1) >>>> go func(){ >>>> defer q.pending.Done() >>>> result := fn() >>>> q.mu.Lock() >>>> q.responses[token] = result >>>> q.mu.Unlock() >>>> }() >>>> return token >>>> } >>>> >>>> func (q *Queries) Wait(){ q.pending.Wait() } >>>> func (q *Queries) Patch(data []byte) []byte { >>>> // replace tokens with responess >>>> } >>>> >>>> func main() { >>>> >>>> q := NewQueries() >>>> >>>> var funcMap = template.FuncMap { >>>> >>>> "sleep": func() Token { return q.Do(func() string { >>>> time.Sleep(1 * time.Second); return "slept" }) }, >>>> >>>> } >>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}} >>>> {{sleep}} {{sleep}}") >>>> var buf bytes.Buffer >>>> >>>> tmpl.Execute(&buf, nil) >>>> >>>> q.Wait() >>>> >>>> os.Stdout.Write(q.Patch(buf.Bytes)) >>>> >>>> } >>>> >>>> The other approach would be to do multiple passes: >>>> >>>> 1. execute template >>>> 2. collect funcs that haven't been run yet >>>> 2.1. no funcs left --> output >>>> 3. execute these funcs, cache the func values >>>> 4. goto step 1 using the cache >>>> >>>> On Wednesday, 31 May 2017 16:26:15 UTC+3, Michael Brown wrote: >>>>> >>>>> I am designing a system that will heavily use text/template processing >>>>> and I've run into one issue that is going to be a show stopper for me if >>>>> I >>>>> can't figure out a way around it. >>>>> >>>>> Execute() on a template will run all of the functions in the template >>>>> serially. >>>>> >>>>> For example, when you run the code below, you can see it output >>>>> "slept" once every second until it completes after 3 seconds. In this >>>>> example, the sleep is simulating an RPC call to another process that may >>>>> take some considerable time (few tenths of a second), but there will be a >>>>> large number of these calls that could all theoretically run in parallel >>>>> (ie. there are no data dependencies between them). I'd really like to >>>>> know >>>>> a way that I could have the templating engine run all of the functions at >>>>> once and collect the output, ie. in the example below, the entire program >>>>> should run in 1 second. >>>>> >>>>> package main >>>>> >>>>> >>>>> import ( >>>>> >>>>> "text/template" >>>>> >>>>> "os" >>>>> >>>>> "time" >>>>> >>>>> ) >>>>> >>>>> >>>>> var funcMap = template.FuncMap { >>>>> >>>>> "sleep": func() string { time.Sleep(1 * time.Second); return >>>>> "slept" }, >>>>> >>>>> } >>>>> >>>>> >>>>> func main() { >>>>> >>>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}} >>>>> {{sleep}} {{sleep}}") >>>>> >>>>> tmpl.Execute(os.Stdout, nil) >>>>> >>>>> } >>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "golang-nuts" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/golang-nuts/j0WwjQE11rw/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> golang-nuts...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.