+1, I think this is a natural extension of the SDF to Python. On Tue, Mar 14, 2017 at 1:19 PM, Chamikara Jayalath <chamik...@apache.org> wrote:
> Thanks Eugene. Will keep you cc'd. > > - Cham > > On Tue, Mar 14, 2017 at 1:15 PM Eugene Kirpichov <kirpic...@google.com> > wrote: > >> Thanks Cham! Agreed that this is a clean way to add SDF API to Python. >> And agreed that it's a good idea to follow as similar path to Java - >> direct runner first. The next easiest thing is going to be the various >> streaming runners. >> Please keep me cc'd on various PRs related to this. >> >> On Tue, Mar 14, 2017 at 12:46 PM Chamikara Jayalath <chamik...@apache.org> >> wrote: >> >> Ccing Eugene and other authors of the original Splittable Dofn proposal. >> >> Thanks, >> Cham >> >> On Mon, Mar 6, 2017 at 10:32 AM Chamikara Jayalath <chamik...@apache.org> >> wrote: >> >> Thanks Ahmet. I created https://issues.apache.org/jira/browse/BEAM-1630 for >> this. >> >> Any further comments are welcome :). >> >> Implementation wise, I think we should add Splittable DoFn to streaming >> direct runner first, after [1] is finalized, and then follow it up with >> support for bounded sources and other runners (currently Dataflow runner >> and Dataflow/other runners through Fn API [2]). >> >> Thanks, >> Cham >> >> [1] https://issues.apache.org/jira/browse/BEAM-1265 >> [2] https://s.apache.org/beam-fn-api >> >> On Fri, Mar 3, 2017 at 5:34 PM Ahmet Altay <al...@google.com.invalid> >> wrote: >> >> +1 Thank you, this is a great and clean API proposal. >> >> Ahmet >> >> On Fri, Mar 3, 2017 at 5:16 PM, Chamikara Jayalath <chamik...@apache.org> >> wrote: >> >> > Hi All, >> > >> > I've put together a document that proposes a Splittable DoFn API for >> Python >> > SDK. >> > >> > https://docs.google.com/document/d/1h_zprJrOilivK2xfvl4L42vaX4DMYGfH >> > 1YDmi-s_ozM/edit?usp=sharing >> > >> > Splittable DoFn framework [1] is currently being implemented for Java >> SDK >> > [2] and will unlock many use-cases that are not possible with the >> current >> > BoundedSource framework [3] (see [1] for details). So, I believe, it >> will >> > be good to add a similar framework to Python SDK as well. >> > >> > Please let me know what you think. >> > >> > Thanks, >> > Cham >> > >> > [1] http://s.apache.org/splittable-do-fn >> > [2] https://issues.apache.org/jira/browse/BEAM-65 >> > [3] >> > https://github.com/apache/beam/blob/master/sdks/python/ >> > apache_beam/io/iobase.py >> > >> >>
