thank you! Shen
On Tue, Aug 15, 2017 at 1:29 PM, Robert Bradshaw < rober...@google.com.invalid> wrote: > On Tue, Aug 15, 2017 at 10:21 AM, Shen Li <cs.she...@gmail.com> wrote: > > Hi Thomas, > > > > Does it mean future Pipeline implementations would allow applications to > > set the runner after a pipeline has been constructed? > > Correct, that's the intent. > > > > > Thanks, > > Shen > > > > On Tue, Aug 15, 2017 at 12:36 PM, Thomas Groh <tg...@google.com.invalid> > > wrote: > > > >> This style of method doesn't fit with the current approach of pipeline > >> construction, where the PipelineRunner need not be specified until the > >> pipeline is run; as such, the runner can't observe the construction of > the > >> Pipeline, as it may not exist during the construction of the Pipeline. > >> > >> On Tue, Aug 15, 2017 at 8:41 AM, Eugene Kirpichov < > >> kirpic...@google.com.invalid> wrote: > >> > >> > ... And remember it and make available inside PCollection (which > >> > application produced this collection). > >> > > >> > On Tue, Aug 15, 2017, 8:39 AM Eugene Kirpichov <kirpic...@google.com> > >> > wrote: > >> > > >> > > In general, no - but the implementation of PAssertionSite > exemplifies > >> the > >> > > approach. I guess it could be useful to make this a general beam > >> feature > >> > > and remember it for all transforms. It would probably be best to > >> > implement > >> > > inside Pipeline.apply(). > >> > > > >> > > On Tue, Aug 15, 2017, 7:02 AM Shen Li <cs.she...@gmail.com> wrote: > >> > > > >> > >> Hi Eugene, > >> > >> > >> > >> Thanks for sharing the info. That PAssertionSite tracks where an > >> > assertion > >> > >> error occurred. Do you know if it is possible to get the class name > >> and > >> > >> line number where a PTransform was added? > >> > >> > >> > >> Thanks, > >> > >> Shen > >> > >> > >> > >> On Mon, Aug 14, 2017 at 10:54 PM, Eugene Kirpichov < > >> > >> kirpic...@google.com.invalid> wrote: > >> > >> > >> > >> > Hi Shen, > >> > >> > Responding just to one part of your message - "remember the line > at > >> > >> which > >> > >> > the PTransform was added": take a look at > >> > >> > https://github.com/apache/beam/pull/2247 which does this for > >> PAssert. > >> > >> > > >> > >> > On Mon, Aug 14, 2017 at 7:32 PM Shen Li <cs.she...@gmail.com> > >> wrote: > >> > >> > > >> > >> > > In 0.5.0 or earlier releases, PipelineRunner provides an > >> > >> > > apply(PTransform<InputT, OutputT>, InputT) method which allows > >> > runner > >> > >> > > implementation to perform actions when the PTransform is added > to > >> > the > >> > >> > > pipeline. In later releases, that apply(...) method has been > >> > replaced > >> > >> by > >> > >> > a > >> > >> > > Pipeline#replaceAll() method, where the runner can only get > >> involved > >> > >> > after > >> > >> > > the pipeline has been fully constructed. In terms of override > >> > >> > PTransforms, > >> > >> > > these two APIs are identical. But, the early API could still be > >> > >> helpful. > >> > >> > > For example, the runner could remember the line at which the > >> > >> PTransform > >> > >> > was > >> > >> > > added, and provide that info to users to assist debugging. Is > it > >> > >> possible > >> > >> > > to add that API back? Or is there any other way to involve the > >> > runner > >> > >> > when > >> > >> > > a PTransform is added? > >> > >> > > > >> > >> > > Thanks, > >> > >> > > Shen > >> > >> > > > >> > >> > > >> > >> > >> > > > >> > > >> >
