From: Chamikara Jayalath <chamik...@google.com> Date: Thu, May 9, 2019 at 7:49 PM To: dev
> From: Maximilian Michels <m...@apache.org> > Date: Thu, May 9, 2019 at 9:21 AM > To: <dev@beam.apache.org> > >> Thanks for sharing your ideas for load testing! >> >> > According to other contributors knowledge/experience: I noticed that >> > streaming with KafkaIO is currently supported by wrapping the >> > ExternalTransform in Python SDK. Do you think that streaming pipelines >> > will "just work" with the current state of portability if I do the same >> > for UnboundedSyntheticSource or is there something else missing? >> >> Basically yes, but it requires a bit more effort than just wrapping >> about ExternalTransform. You need to provide an ExternalTransformBuilder >> for the transform to be configured externally. >> >> In portability UnboundedSources can only be supported via SDF. To still >> be able to use legacy IO which uses UnboundedSource the Runner has to >> supply this capability (which the Flink Runner supports). This will >> likely go away if we have an UnboundedSource SDF Wrapper :) > > > So, FlinkRunner has some sort of special support for executing > UnboundedSource via the runner in the portable world ? I see a transform > override for bounded sources in PortableRunner [1] but nothing for unbounded > sources. > > Agree, that we cannot properly support cross-language unbounded sources till > we have SDF and a proper unbounded source to SDF wrapper. That is correct. Go will need SDF support as well. As waiting on implementing the expansion service, except for the vending of extra artifacts (which will be an extension), we discussed this earlier and it's considered stable and ready to build on now.
