Eugene Kirpichov created BEAM-1824:
--------------------------------------
Summary: Adapter for running SDF on a statically known input as a
BoundedSource
Key: BEAM-1824
URL: https://issues.apache.org/jira/browse/BEAM-1824
Project: Beam
Issue Type: New Feature
Components: runner-dataflow, sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov
[~bchambers] suggested the following idea: while the runner implementation of
SDF [BEAM-65] is not yet complete enough to support dynamic rebalancing
(especially over the Fn API), we can special-case the case of Create.of(single
input) + ParDo(SDF) by running it via BoundedSource.
This will allow us to start transitioning bounded IO connectors to SDF API
while preserving the dynamic rebalancing feature in the common case when the
source is known at pipeline submission time.
And then, when SDF runner support catches up, we'll simply add APIs to the IO
connectors for reading from a PCollection of inputs, and those will enjoy the
same benefits. Actually we can add such APIs earlier, with the caveat that they
won't support dynamic rebalancing, but in this case it's ok because there'll be
no performance regression because these APIs didn't exist before.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
