Using impulse is a precursor for both bounded and unbounded SDF. This JIRA represents the work that would be to add support for unbounded SDF using portability APIs: https://issues.apache.org/jira/browse/BEAM-2939
On Fri, Mar 23, 2018 at 11:46 AM Thomas Weise <t...@apache.org> wrote: > So for streaming, we will need the Impulse translation for bounded input, > identical with batch, and then in addition to that support for SDF? > > Any pointers what's involved in adding the SDF support? Is it runner > specific? Does the ULR cover it? > > > On Fri, Mar 23, 2018 at 11:26 AM, Lukasz Cwik <lc...@google.com> wrote: > >> All "sources" in portability will use splittable DoFns for execution. >> >> Specifically, runners will need to be able to checkpoint unbounded >> sources to get a minimum viable pipeline working. >> For bounded pipelines, a DoFn can read the contents of a bounded source. >> >> >> On Fri, Mar 23, 2018 at 10:52 AM Thomas Weise <t...@apache.org> wrote: >> >>> Hi, >>> >>> I'm looking at the portable pipeline translation for streaming. I >>> understand that for batch pipelines, it is sufficient to translate Impulse. >>> >>> What is the intended path to support unbounded sources? >>> >>> The goal here is to get a minimum translation working that will allow >>> streaming wordcount execution. >>> >>> Thanks, >>> Thomas >>> >>> >
