Hi Luke, Great to hear about such progress on this!
Talking about opt-out for all runners in the future, will it require any code changes for current “*Source”-based IOs or the wrappers should completely smooth this transition? Do we need to require to create new IOs only based on SDF or again, the wrappers should help to avoid this? > On 10 Aug 2020, at 22:59, Luke Cwik <lc...@google.com> wrote: > > In the past couple of months wrappers[1, 2] have been added to the Beam Java > SDK which can execute BoundedSource and UnboundedSource as Splittable DoFns. > These have been opt-out for portable pipelines (e.g. Dataflow runner v2, > XLang pipelines on Flink/Spark) and opt-in using an experiment for all other > pipelines. > > I would like to start making the non-portable pipelines starting with the > DirectRunner[3] to be opt-out with the plan that eventually all runners will > only execute splittable DoFns and the BoundedSource/UnboundedSource specific > execution logic from the runners will be removed. > > Users will be able to opt-in any pipeline using the experiment 'use_sdf_read' > and opt-out with the experiment 'use_deprecated_read'. (For portable > pipelines these experiments were 'beam_fn_api' and > 'beam_fn_api_use_deprecated_read' respectively and I have added these two > additional aliases to make the experience less confusing). > > 1: > https://github.com/apache/beam/blob/af1ce643d8fde5352d4519a558de4a2dfd24721d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L275 > > <https://github.com/apache/beam/blob/af1ce643d8fde5352d4519a558de4a2dfd24721d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L275> > 2: > https://github.com/apache/beam/blob/af1ce643d8fde5352d4519a558de4a2dfd24721d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L449 > > <https://github.com/apache/beam/blob/af1ce643d8fde5352d4519a558de4a2dfd24721d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L449> > 3: https://github.com/apache/beam/pull/12519 > <https://github.com/apache/beam/pull/12519>
