Hi Thomas, Does it mean future Pipeline implementations would allow applications to set the runner after a pipeline has been constructed?
Thanks, Shen On Tue, Aug 15, 2017 at 12:36 PM, Thomas Groh <tg...@google.com.invalid> wrote: > This style of method doesn't fit with the current approach of pipeline > construction, where the PipelineRunner need not be specified until the > pipeline is run; as such, the runner can't observe the construction of the > Pipeline, as it may not exist during the construction of the Pipeline. > > On Tue, Aug 15, 2017 at 8:41 AM, Eugene Kirpichov < > kirpic...@google.com.invalid> wrote: > > > ... And remember it and make available inside PCollection (which > > application produced this collection). > > > > On Tue, Aug 15, 2017, 8:39 AM Eugene Kirpichov <kirpic...@google.com> > > wrote: > > > > > In general, no - but the implementation of PAssertionSite exemplifies > the > > > approach. I guess it could be useful to make this a general beam > feature > > > and remember it for all transforms. It would probably be best to > > implement > > > inside Pipeline.apply(). > > > > > > On Tue, Aug 15, 2017, 7:02 AM Shen Li <cs.she...@gmail.com> wrote: > > > > > >> Hi Eugene, > > >> > > >> Thanks for sharing the info. That PAssertionSite tracks where an > > assertion > > >> error occurred. Do you know if it is possible to get the class name > and > > >> line number where a PTransform was added? > > >> > > >> Thanks, > > >> Shen > > >> > > >> On Mon, Aug 14, 2017 at 10:54 PM, Eugene Kirpichov < > > >> kirpic...@google.com.invalid> wrote: > > >> > > >> > Hi Shen, > > >> > Responding just to one part of your message - "remember the line at > > >> which > > >> > the PTransform was added": take a look at > > >> > https://github.com/apache/beam/pull/2247 which does this for > PAssert. > > >> > > > >> > On Mon, Aug 14, 2017 at 7:32 PM Shen Li <cs.she...@gmail.com> > wrote: > > >> > > > >> > > In 0.5.0 or earlier releases, PipelineRunner provides an > > >> > > apply(PTransform<InputT, OutputT>, InputT) method which allows > > runner > > >> > > implementation to perform actions when the PTransform is added to > > the > > >> > > pipeline. In later releases, that apply(...) method has been > > replaced > > >> by > > >> > a > > >> > > Pipeline#replaceAll() method, where the runner can only get > involved > > >> > after > > >> > > the pipeline has been fully constructed. In terms of override > > >> > PTransforms, > > >> > > these two APIs are identical. But, the early API could still be > > >> helpful. > > >> > > For example, the runner could remember the line at which the > > >> PTransform > > >> > was > > >> > > added, and provide that info to users to assist debugging. Is it > > >> possible > > >> > > to add that API back? Or is there any other way to involve the > > runner > > >> > when > > >> > > a PTransform is added? > > >> > > > > >> > > Thanks, > > >> > > Shen > > >> > > > > >> > > > >> > > > > > >
