And after I re-read your answer, I realize that I probably just restated very badly what you wrote, Egon. Thanks! I'll have some time this evening to hack on it.
On Wednesday, May 31, 2017 at 2:51:51 PM UTC-5, Michael Brown wrote: > > Ah! I think I have the kernel of a usable idea here. Thanks. > > How does this sound: template 1 outputs template 2. Input to template 1 > kicks off the goroutines and places {{ get_results token_xyz }} as the > output, but also has an output channel that we can wait on for all of the > answers Then, template 2 is executed with a function that pulls the output > from the channel. Sort of hard to explain, but seems on the face to be > workable. I'll post results after I prototype it. > > -- > Michael > > On Wednesday, May 31, 2017 at 1:23:40 PM UTC-5, Egon wrote: >> >> On Wednesday, 31 May 2017 20:34:56 UTC+3, Michael Brown wrote: >>> >>> This is almost certainly going to be a case where you have the correct >>> solution and I don't have the go experience to properly understand it. I >>> wasn't quite understanding how your code was "patching" the results and how >>> the template package knows to wait for it. >>> >> >> Let's say you have a template >> >> {{ sleep }} xxx {{ sleep }} >> >> Once you render with that approach you get something like this... >> >> <<token1>> xxx <<token2>> >> >> So instead of returning the actual value you replace them with a token. >> And spawn a go routine to fetch the answers. >> >> You wait for the results or just some, doesn't matter. >> >> Once you get a results, you replace it in the buffer... >> >> token1 -> "alpha" >> token2 -> "beta" >> >> "alpha" xxx "beta" >> >> Or whatever the result is. >> >> However this approach is pretty limited when you want to range over the >> results. >> >> Doing two passes over the template, might work better, but needs somewhat >> better logic to handle lookup; but that could work for handling multiple >> results... >> >> {{ sleep }} >> >> In the first pass you start the goroutines and each output goes into an >> array of channels >> >> var rpcs = []func() interface{} >> >> var funcMap = template.FuncMap { "sleep": func() { rpcs = append(rpcs, >> sleep) }, } >> >> // now you schedule goroutines in whatever way you need for the rpcs and >> write the results to a channel >> var results = []chan interface{} >> >> the second pass would do it as >> current := 0 >> >> var funcMap = template.FuncMap { "sleep": func() interface{} { >> >> result := <-results[current] >> >> current++ >> >> return result >> >> >> // ... >> >> >> Without knowing the actual template, it is hard to recommend something >> better simpler or easier. >> >> Let me describe my problem. >>> >>> I have a REST interface running on an embedded system. The REST api is >>> mandated by another entity and I have zero control over the output, I have >>> to produce specific conformant output. The data that I need to build the >>> interface is on the system in a variety of other processes, and I can get >>> that data via a couple of different RPC mechanisms, however sometimes these >>> RPC mechanisms can be slow. (I'm in progress building go bindings for them). >>> >>> The current code which creates the REST interface is a huge morass of C >>> code that generates the JSON output, so the exact structure of the JSON >>> output is hardcoded in the code. >>> >>> I have an opportunity here, and that is that it appears to me that, with >>> a little work, I can completely templatize the rest output. That is, I can >>> have zero page-specific code and render almost all of the output by >>> providing a small number of generic data access functions. The issue I ran >>> into is the serial nature of the substitution kills the performance in my >>> prototype. >>> >>> Yes, I can make a specific go function per page that gathers all the >>> data I need and provides it to the template. But that would mean that I'd >>> need to maintain both the templates and the data access functions. This is >>> still an improvement on the old way of doing things, but I was hoping to >>> jump straight to a fully templatized system, which according to my initial >>> analysis, would be considerably less code and would not really be abusing >>> the template system (most of the function calls are very straightforward >>> and there is no heavy processing of the output). >>> >>> -- >>> Michael >>> >>> On Wednesday, May 31, 2017 at 10:56:57 AM UTC-5, Egon wrote: >>>> >>>> Both of my described approaches run the funcs serially however it does >>>> not wait for the response and later patxhes the results. >>>> >>>> Can you describe the whole thing you are building? Piecing the >>>> requirements and purpose together from comments is difficult. >>>> >>>> I.e. how much memory, how big is request latency, who gets the output, >>>> where do the templates come from... >>>> >>>> On May 31, 2017 6:40 PM, "Michael Brown" <michael...@gmail.com> wrote: >>>> >>>> The best thing I can think of is to modify text/template to add futures >>>> support, and then do multi-pass rendering. The place to add this looks >>>> relatively simple, however the implementation looks complicated (and I >>>> just >>>> wrote my first program in go a couple weeks ago...) >>>> >>>> The problem that I see with the solution below is that text/template >>>> execution tries to instantiate the values of each function serially and >>>> essentially waits for completion on each, serially. I've spent the last >>>> couple hours examining the text/template implementation. >>>> -- >>>> Michael >>>> >>>> >>>> On Wednesday, May 31, 2017 at 8:59:20 AM UTC-5, Egon wrote: >>>>> >>>>> The best idea I can think of (without digging and modifying >>>>> text/template) is to use special tokens and replace them afterwards... >>>>> Of course this approach has limitations in what it can do. >>>>> >>>>> *// note: code untested and incomplete* >>>>> >>>>> type Token string >>>>> type Queries struct { >>>>> pending sync.WaitGroup >>>>> mu sync.Mutex >>>>> responses map[Token]string >>>>> } >>>>> >>>>> func (q *Queries) createToken() Token { >>>>> return unique token >>>>> } >>>>> >>>>> func (q *Queries) Do(fn func() string) Token { >>>>> token := q.createToken() >>>>> q.pending.Add(1) >>>>> go func(){ >>>>> defer q.pending.Done() >>>>> result := fn() >>>>> q.mu.Lock() >>>>> q.responses[token] = result >>>>> q.mu.Unlock() >>>>> }() >>>>> return token >>>>> } >>>>> >>>>> func (q *Queries) Wait(){ q.pending.Wait() } >>>>> func (q *Queries) Patch(data []byte) []byte { >>>>> // replace tokens with responess >>>>> } >>>>> >>>>> func main() { >>>>> >>>>> q := NewQueries() >>>>> >>>>> var funcMap = template.FuncMap { >>>>> >>>>> "sleep": func() Token { return q.Do(func() string { >>>>> time.Sleep(1 * time.Second); return "slept" }) }, >>>>> >>>>> } >>>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}} >>>>> {{sleep}} {{sleep}}") >>>>> var buf bytes.Buffer >>>>> >>>>> tmpl.Execute(&buf, nil) >>>>> >>>>> q.Wait() >>>>> >>>>> os.Stdout.Write(q.Patch(buf.Bytes)) >>>>> >>>>> } >>>>> >>>>> The other approach would be to do multiple passes: >>>>> >>>>> 1. execute template >>>>> 2. collect funcs that haven't been run yet >>>>> 2.1. no funcs left --> output >>>>> 3. execute these funcs, cache the func values >>>>> 4. goto step 1 using the cache >>>>> >>>>> On Wednesday, 31 May 2017 16:26:15 UTC+3, Michael Brown wrote: >>>>>> >>>>>> I am designing a system that will heavily use text/template >>>>>> processing and I've run into one issue that is going to be a show >>>>>> stopper >>>>>> for me if I can't figure out a way around it. >>>>>> >>>>>> Execute() on a template will run all of the functions in the template >>>>>> serially. >>>>>> >>>>>> For example, when you run the code below, you can see it output >>>>>> "slept" once every second until it completes after 3 seconds. In this >>>>>> example, the sleep is simulating an RPC call to another process that may >>>>>> take some considerable time (few tenths of a second), but there will be >>>>>> a >>>>>> large number of these calls that could all theoretically run in parallel >>>>>> (ie. there are no data dependencies between them). I'd really like to >>>>>> know >>>>>> a way that I could have the templating engine run all of the functions >>>>>> at >>>>>> once and collect the output, ie. in the example below, the entire >>>>>> program >>>>>> should run in 1 second. >>>>>> >>>>>> package main >>>>>> >>>>>> >>>>>> import ( >>>>>> >>>>>> "text/template" >>>>>> >>>>>> "os" >>>>>> >>>>>> "time" >>>>>> >>>>>> ) >>>>>> >>>>>> >>>>>> var funcMap = template.FuncMap { >>>>>> >>>>>> "sleep": func() string { time.Sleep(1 * time.Second); return >>>>>> "slept" }, >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> func main() { >>>>>> >>>>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}} >>>>>> {{sleep}} {{sleep}}") >>>>>> >>>>>> tmpl.Execute(os.Stdout, nil) >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "golang-nuts" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/golang-nuts/j0WwjQE11rw/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> golang-nuts...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.