Thanks

On Thu, Jun 13, 2019 at 2:02 PM Łukasz Gajowy <lgaj...@apache.org> wrote:
>
> Created a PR: https://github.com/apache/beam/pull/8846
>
> śr., 12 cze 2019 o 11:40 Ismaël Mejía <ieme...@gmail.com> napisał(a):
>>
>> Can you please add this to the design documents webpage.
>> https://beam.apache.org/contribute/design-documents/
>>
>> On Fri, May 10, 2019 at 11:50 AM Maximilian Michels <m...@apache.org> wrote:
>> >
>> > > So, FlinkRunner has some sort of special support for executing 
>> > > UnboundedSource via the runner in the portable world ? I see a transform 
>> > > override for bounded sources in PortableRunner [1] but nothing for 
>> > > unbounded sources.
>> >
>> > It's in the translation code:
>> > https://github.com/apache/beam/blob/6679b00138a5b82a6a55e7bc94c453957cea501c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java#L216
>> >
>> > For migration I think that's a valid path, especially because Runners
>> > already have the translation code in place. We can later swap-out the
>> > UnboundedSource translation with the SDF wrapper.
>> >
>> > -Max
>> >
>> > On 09.05.19 22:46, Robert Bradshaw wrote:
>> > > From: Chamikara Jayalath <chamik...@google.com>
>> > > Date: Thu, May 9, 2019 at 7:49 PM
>> > > To: dev
>> > >
>> > >> From: Maximilian Michels <m...@apache.org>
>> > >> Date: Thu, May 9, 2019 at 9:21 AM
>> > >> To: <dev@beam.apache.org>
>> > >>
>> > >>> Thanks for sharing your ideas for load testing!
>> > >>>
>> > >>>> According to other contributors knowledge/experience: I noticed that 
>> > >>>> streaming with KafkaIO is currently supported by wrapping the 
>> > >>>> ExternalTransform in Python SDK. Do you think that streaming 
>> > >>>> pipelines will "just work" with the current state of portability if I 
>> > >>>> do the same for UnboundedSyntheticSource or is there something else 
>> > >>>> missing?
>> > >>>
>> > >>> Basically yes, but it requires a bit more effort than just wrapping
>> > >>> about ExternalTransform. You need to provide an 
>> > >>> ExternalTransformBuilder
>> > >>> for the transform to be configured externally.
>> > >>>
>> > >>> In portability UnboundedSources can only be supported via SDF. To still
>> > >>> be able to use legacy IO which uses UnboundedSource the Runner has to
>> > >>> supply this capability (which the Flink Runner supports). This will
>> > >>> likely go away if we have an UnboundedSource SDF Wrapper :)
>> > >>
>> > >>
>> > >> So, FlinkRunner has some sort of special support for executing 
>> > >> UnboundedSource via the runner in the portable world ? I see a 
>> > >> transform override for bounded sources in PortableRunner [1] but 
>> > >> nothing for unbounded sources.
>> > >>
>> > >> Agree, that we cannot properly support cross-language unbounded sources 
>> > >> till we have SDF and a proper unbounded source to SDF wrapper.
>> > >
>> > > That is correct. Go will need SDF support as well.
>> > >
>> > > As waiting on implementing the expansion service, except for the
>> > > vending of extra artifacts (which will be an extension), we discussed
>> > > this earlier and it's considered stable and ready to build on now.
>> > >

Reply via email to