Can you please add this to the design documents webpage.
https://beam.apache.org/contribute/design-documents/

On Fri, May 10, 2019 at 11:50 AM Maximilian Michels <m...@apache.org> wrote:
>
> > So, FlinkRunner has some sort of special support for executing 
> > UnboundedSource via the runner in the portable world ? I see a transform 
> > override for bounded sources in PortableRunner [1] but nothing for 
> > unbounded sources.
>
> It's in the translation code:
> https://github.com/apache/beam/blob/6679b00138a5b82a6a55e7bc94c453957cea501c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java#L216
>
> For migration I think that's a valid path, especially because Runners
> already have the translation code in place. We can later swap-out the
> UnboundedSource translation with the SDF wrapper.
>
> -Max
>
> On 09.05.19 22:46, Robert Bradshaw wrote:
> > From: Chamikara Jayalath <chamik...@google.com>
> > Date: Thu, May 9, 2019 at 7:49 PM
> > To: dev
> >
> >> From: Maximilian Michels <m...@apache.org>
> >> Date: Thu, May 9, 2019 at 9:21 AM
> >> To: <dev@beam.apache.org>
> >>
> >>> Thanks for sharing your ideas for load testing!
> >>>
> >>>> According to other contributors knowledge/experience: I noticed that 
> >>>> streaming with KafkaIO is currently supported by wrapping the 
> >>>> ExternalTransform in Python SDK. Do you think that streaming pipelines 
> >>>> will "just work" with the current state of portability if I do the same 
> >>>> for UnboundedSyntheticSource or is there something else missing?
> >>>
> >>> Basically yes, but it requires a bit more effort than just wrapping
> >>> about ExternalTransform. You need to provide an ExternalTransformBuilder
> >>> for the transform to be configured externally.
> >>>
> >>> In portability UnboundedSources can only be supported via SDF. To still
> >>> be able to use legacy IO which uses UnboundedSource the Runner has to
> >>> supply this capability (which the Flink Runner supports). This will
> >>> likely go away if we have an UnboundedSource SDF Wrapper :)
> >>
> >>
> >> So, FlinkRunner has some sort of special support for executing 
> >> UnboundedSource via the runner in the portable world ? I see a transform 
> >> override for bounded sources in PortableRunner [1] but nothing for 
> >> unbounded sources.
> >>
> >> Agree, that we cannot properly support cross-language unbounded sources 
> >> till we have SDF and a proper unbounded source to SDF wrapper.
> >
> > That is correct. Go will need SDF support as well.
> >
> > As waiting on implementing the expansion service, except for the
> > vending of extra artifacts (which will be an extension), we discussed
> > this earlier and it's considered stable and ready to build on now.
> >

Reply via email to