Thanks
On Thu, Jun 13, 2019 at 2:02 PM Łukasz Gajowy <lgaj...@apache.org> wrote: > > Created a PR: https://github.com/apache/beam/pull/8846 > > śr., 12 cze 2019 o 11:40 Ismaël Mejía <ieme...@gmail.com> napisał(a): >> >> Can you please add this to the design documents webpage. >> https://beam.apache.org/contribute/design-documents/ >> >> On Fri, May 10, 2019 at 11:50 AM Maximilian Michels <m...@apache.org> wrote: >> > >> > > So, FlinkRunner has some sort of special support for executing >> > > UnboundedSource via the runner in the portable world ? I see a transform >> > > override for bounded sources in PortableRunner [1] but nothing for >> > > unbounded sources. >> > >> > It's in the translation code: >> > https://github.com/apache/beam/blob/6679b00138a5b82a6a55e7bc94c453957cea501c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java#L216 >> > >> > For migration I think that's a valid path, especially because Runners >> > already have the translation code in place. We can later swap-out the >> > UnboundedSource translation with the SDF wrapper. >> > >> > -Max >> > >> > On 09.05.19 22:46, Robert Bradshaw wrote: >> > > From: Chamikara Jayalath <chamik...@google.com> >> > > Date: Thu, May 9, 2019 at 7:49 PM >> > > To: dev >> > > >> > >> From: Maximilian Michels <m...@apache.org> >> > >> Date: Thu, May 9, 2019 at 9:21 AM >> > >> To: <dev@beam.apache.org> >> > >> >> > >>> Thanks for sharing your ideas for load testing! >> > >>> >> > >>>> According to other contributors knowledge/experience: I noticed that >> > >>>> streaming with KafkaIO is currently supported by wrapping the >> > >>>> ExternalTransform in Python SDK. Do you think that streaming >> > >>>> pipelines will "just work" with the current state of portability if I >> > >>>> do the same for UnboundedSyntheticSource or is there something else >> > >>>> missing? >> > >>> >> > >>> Basically yes, but it requires a bit more effort than just wrapping >> > >>> about ExternalTransform. You need to provide an >> > >>> ExternalTransformBuilder >> > >>> for the transform to be configured externally. >> > >>> >> > >>> In portability UnboundedSources can only be supported via SDF. To still >> > >>> be able to use legacy IO which uses UnboundedSource the Runner has to >> > >>> supply this capability (which the Flink Runner supports). This will >> > >>> likely go away if we have an UnboundedSource SDF Wrapper :) >> > >> >> > >> >> > >> So, FlinkRunner has some sort of special support for executing >> > >> UnboundedSource via the runner in the portable world ? I see a >> > >> transform override for bounded sources in PortableRunner [1] but >> > >> nothing for unbounded sources. >> > >> >> > >> Agree, that we cannot properly support cross-language unbounded sources >> > >> till we have SDF and a proper unbounded source to SDF wrapper. >> > > >> > > That is correct. Go will need SDF support as well. >> > > >> > > As waiting on implementing the expansion service, except for the >> > > vending of extra artifacts (which will be an extension), we discussed >> > > this earlier and it's considered stable and ready to build on now. >> > >