Can you please add this to the design documents webpage. https://beam.apache.org/contribute/design-documents/
On Fri, May 10, 2019 at 11:50 AM Maximilian Michels <m...@apache.org> wrote: > > > So, FlinkRunner has some sort of special support for executing > > UnboundedSource via the runner in the portable world ? I see a transform > > override for bounded sources in PortableRunner [1] but nothing for > > unbounded sources. > > It's in the translation code: > https://github.com/apache/beam/blob/6679b00138a5b82a6a55e7bc94c453957cea501c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java#L216 > > For migration I think that's a valid path, especially because Runners > already have the translation code in place. We can later swap-out the > UnboundedSource translation with the SDF wrapper. > > -Max > > On 09.05.19 22:46, Robert Bradshaw wrote: > > From: Chamikara Jayalath <chamik...@google.com> > > Date: Thu, May 9, 2019 at 7:49 PM > > To: dev > > > >> From: Maximilian Michels <m...@apache.org> > >> Date: Thu, May 9, 2019 at 9:21 AM > >> To: <dev@beam.apache.org> > >> > >>> Thanks for sharing your ideas for load testing! > >>> > >>>> According to other contributors knowledge/experience: I noticed that > >>>> streaming with KafkaIO is currently supported by wrapping the > >>>> ExternalTransform in Python SDK. Do you think that streaming pipelines > >>>> will "just work" with the current state of portability if I do the same > >>>> for UnboundedSyntheticSource or is there something else missing? > >>> > >>> Basically yes, but it requires a bit more effort than just wrapping > >>> about ExternalTransform. You need to provide an ExternalTransformBuilder > >>> for the transform to be configured externally. > >>> > >>> In portability UnboundedSources can only be supported via SDF. To still > >>> be able to use legacy IO which uses UnboundedSource the Runner has to > >>> supply this capability (which the Flink Runner supports). This will > >>> likely go away if we have an UnboundedSource SDF Wrapper :) > >> > >> > >> So, FlinkRunner has some sort of special support for executing > >> UnboundedSource via the runner in the portable world ? I see a transform > >> override for bounded sources in PortableRunner [1] but nothing for > >> unbounded sources. > >> > >> Agree, that we cannot properly support cross-language unbounded sources > >> till we have SDF and a proper unbounded source to SDF wrapper. > > > > That is correct. Go will need SDF support as well. > > > > As waiting on implementing the expansion service, except for the > > vending of extra artifacts (which will be an extension), we discussed > > this earlier and it's considered stable and ready to build on now. > >