On Tue, Aug 15, 2017 at 10:21 AM, Shen Li <cs.she...@gmail.com> wrote: > Hi Thomas, > > Does it mean future Pipeline implementations would allow applications to > set the runner after a pipeline has been constructed?
Correct, that's the intent. > > Thanks, > Shen > > On Tue, Aug 15, 2017 at 12:36 PM, Thomas Groh <tg...@google.com.invalid> > wrote: > >> This style of method doesn't fit with the current approach of pipeline >> construction, where the PipelineRunner need not be specified until the >> pipeline is run; as such, the runner can't observe the construction of the >> Pipeline, as it may not exist during the construction of the Pipeline. >> >> On Tue, Aug 15, 2017 at 8:41 AM, Eugene Kirpichov < >> kirpic...@google.com.invalid> wrote: >> >> > ... And remember it and make available inside PCollection (which >> > application produced this collection). >> > >> > On Tue, Aug 15, 2017, 8:39 AM Eugene Kirpichov <kirpic...@google.com> >> > wrote: >> > >> > > In general, no - but the implementation of PAssertionSite exemplifies >> the >> > > approach. I guess it could be useful to make this a general beam >> feature >> > > and remember it for all transforms. It would probably be best to >> > implement >> > > inside Pipeline.apply(). >> > > >> > > On Tue, Aug 15, 2017, 7:02 AM Shen Li <cs.she...@gmail.com> wrote: >> > > >> > >> Hi Eugene, >> > >> >> > >> Thanks for sharing the info. That PAssertionSite tracks where an >> > assertion >> > >> error occurred. Do you know if it is possible to get the class name >> and >> > >> line number where a PTransform was added? >> > >> >> > >> Thanks, >> > >> Shen >> > >> >> > >> On Mon, Aug 14, 2017 at 10:54 PM, Eugene Kirpichov < >> > >> kirpic...@google.com.invalid> wrote: >> > >> >> > >> > Hi Shen, >> > >> > Responding just to one part of your message - "remember the line at >> > >> which >> > >> > the PTransform was added": take a look at >> > >> > https://github.com/apache/beam/pull/2247 which does this for >> PAssert. >> > >> > >> > >> > On Mon, Aug 14, 2017 at 7:32 PM Shen Li <cs.she...@gmail.com> >> wrote: >> > >> > >> > >> > > In 0.5.0 or earlier releases, PipelineRunner provides an >> > >> > > apply(PTransform<InputT, OutputT>, InputT) method which allows >> > runner >> > >> > > implementation to perform actions when the PTransform is added to >> > the >> > >> > > pipeline. In later releases, that apply(...) method has been >> > replaced >> > >> by >> > >> > a >> > >> > > Pipeline#replaceAll() method, where the runner can only get >> involved >> > >> > after >> > >> > > the pipeline has been fully constructed. In terms of override >> > >> > PTransforms, >> > >> > > these two APIs are identical. But, the early API could still be >> > >> helpful. >> > >> > > For example, the runner could remember the line at which the >> > >> PTransform >> > >> > was >> > >> > > added, and provide that info to users to assist debugging. Is it >> > >> possible >> > >> > > to add that API back? Or is there any other way to involve the >> > runner >> > >> > when >> > >> > > a PTransform is added? >> > >> > > >> > >> > > Thanks, >> > >> > > Shen >> > >> > > >> > >> > >> > >> >> > > >> > >>
