+1 I’m a bit hesitant, though, because stage 3 (in the new plan) could become the current stage 1: now, stage 1 is “waiting for Runners to support SDF” while stage 2 is “implement sources as SDF”. We are blocked by Runner support in stage 1 while in the new scheme we would be blocked on Runner support only in stage 3. Also, we introduce the additional burden of implementing immediate SDF implementations, i.e. in the old scheme we go source API -> SDF while in the new scheme we go source API -> “SDF but not really because Runners don’t support it” -> SDF for real.
— Aljoscha > On 1. May 2017, at 07:22, Eugene Kirpichov <[email protected]> > wrote: > > Hey all, > > TL;DR: Development of SDF ecosystem (transitioning existing connectors to > SDF, building libraries, battle-testing the API) is currently blocked on > having SDF supported at full parity with Source API in all Beam runners, > which will take a long time. > > But we can unblock all this work and start doing it very soon, by running a > special case of SDF on top of the Source API. > > I think this is very exciting. Please comment on the following short > proposal! > > https://s.apache.org/sdf-via-source > > After getting some consensus on the doc and in this thread, I will start > filing a network of JIRAs and follow up with next steps.
