Created a PR: https://github.com/apache/beam/pull/8846
śr., 12 cze 2019 o 11:40 Ismaël Mejía <ieme...@gmail.com> napisał(a): > Can you please add this to the design documents webpage. > https://beam.apache.org/contribute/design-documents/ > > On Fri, May 10, 2019 at 11:50 AM Maximilian Michels <m...@apache.org> > wrote: > > > > > So, FlinkRunner has some sort of special support for executing > UnboundedSource via the runner in the portable world ? I see a transform > override for bounded sources in PortableRunner [1] but nothing for > unbounded sources. > > > > It's in the translation code: > > > https://github.com/apache/beam/blob/6679b00138a5b82a6a55e7bc94c453957cea501c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java#L216 > > > > For migration I think that's a valid path, especially because Runners > > already have the translation code in place. We can later swap-out the > > UnboundedSource translation with the SDF wrapper. > > > > -Max > > > > On 09.05.19 22:46, Robert Bradshaw wrote: > > > From: Chamikara Jayalath <chamik...@google.com> > > > Date: Thu, May 9, 2019 at 7:49 PM > > > To: dev > > > > > >> From: Maximilian Michels <m...@apache.org> > > >> Date: Thu, May 9, 2019 at 9:21 AM > > >> To: <dev@beam.apache.org> > > >> > > >>> Thanks for sharing your ideas for load testing! > > >>> > > >>>> According to other contributors knowledge/experience: I noticed > that streaming with KafkaIO is currently supported by wrapping the > ExternalTransform in Python SDK. Do you think that streaming pipelines will > "just work" with the current state of portability if I do the same for > UnboundedSyntheticSource or is there something else missing? > > >>> > > >>> Basically yes, but it requires a bit more effort than just wrapping > > >>> about ExternalTransform. You need to provide an > ExternalTransformBuilder > > >>> for the transform to be configured externally. > > >>> > > >>> In portability UnboundedSources can only be supported via SDF. To > still > > >>> be able to use legacy IO which uses UnboundedSource the Runner has to > > >>> supply this capability (which the Flink Runner supports). This will > > >>> likely go away if we have an UnboundedSource SDF Wrapper :) > > >> > > >> > > >> So, FlinkRunner has some sort of special support for executing > UnboundedSource via the runner in the portable world ? I see a transform > override for bounded sources in PortableRunner [1] but nothing for > unbounded sources. > > >> > > >> Agree, that we cannot properly support cross-language unbounded > sources till we have SDF and a proper unbounded source to SDF wrapper. > > > > > > That is correct. Go will need SDF support as well. > > > > > > As waiting on implementing the expansion service, except for the > > > vending of extra artifacts (which will be an extension), we discussed > > > this earlier and it's considered stable and ready to build on now. > > > >
