So, FlinkRunner has some sort of special support for executing UnboundedSource
via the runner in the portable world ? I see a transform override for bounded
sources in PortableRunner [1] but nothing for unbounded sources.
It's in the translation code:
https://github.com/apache/beam/blob/6679b00138a5b82a6a55e7bc94c453957cea501c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java#L216
For migration I think that's a valid path, especially because Runners
already have the translation code in place. We can later swap-out the
UnboundedSource translation with the SDF wrapper.
-Max
On 09.05.19 22:46, Robert Bradshaw wrote:
From: Chamikara Jayalath <chamik...@google.com>
Date: Thu, May 9, 2019 at 7:49 PM
To: dev
From: Maximilian Michels <m...@apache.org>
Date: Thu, May 9, 2019 at 9:21 AM
To: <dev@beam.apache.org>
Thanks for sharing your ideas for load testing!
According to other contributors knowledge/experience: I noticed that streaming with
KafkaIO is currently supported by wrapping the ExternalTransform in Python SDK. Do you
think that streaming pipelines will "just work" with the current state of
portability if I do the same for UnboundedSyntheticSource or is there something else
missing?
Basically yes, but it requires a bit more effort than just wrapping
about ExternalTransform. You need to provide an ExternalTransformBuilder
for the transform to be configured externally.
In portability UnboundedSources can only be supported via SDF. To still
be able to use legacy IO which uses UnboundedSource the Runner has to
supply this capability (which the Flink Runner supports). This will
likely go away if we have an UnboundedSource SDF Wrapper :)
So, FlinkRunner has some sort of special support for executing UnboundedSource
via the runner in the portable world ? I see a transform override for bounded
sources in PortableRunner [1] but nothing for unbounded sources.
Agree, that we cannot properly support cross-language unbounded sources till we
have SDF and a proper unbounded source to SDF wrapper.
That is correct. Go will need SDF support as well.
As waiting on implementing the expansion service, except for the
vending of extra artifacts (which will be an extension), we discussed
this earlier and it's considered stable and ready to build on now.
