The three github repos I mentioned will all be contributed. That means that we plan to have Runner implementations for Cloud Dataflow, Spark, and Flink included in the Beam project. Note however, that unlike the DirectRunner each of these call out to significant chunks of code located elsewhere (Cloud Dataflow service, Apache Flink, Apache Spark).
Perhaps we need a better term to refer to Runners that clarify it may delegate a large portion of functionality to an existing underlying system. On Wed, Feb 10, 2016 at 5:54 PM, P. Taylor Goetz <[email protected]> wrote: > Good question, and not clear from the proposal. > > Probably a good first thing (among many) for the podling community to > figure out... > > -Taylor > > > On Feb 10, 2016, at 8:11 PM, Henry Saputra <[email protected]> > wrote: > > > > Hi Frances, > > > > Just want to make sure, the plan is to make existing Runners to be > brought > > in into Bean source code repository as internal modules, right? > > > > Thanks, > > > > Henry > > > > On Wed, Feb 10, 2016 at 4:59 PM, Frances Perry <[email protected]> > > wrote: > > > >> One of the initial things we are focusing on as we move things to Beam > is > >> cleaning up the API that new runners will implement. (The current > version > >> is neither particularly stable nor clean.) But as long as you are > >> comfortable with a little churn, feel free to start poking around in the > >> code that will be seeding the repository (after some in-progress > >> refactoring): > >> > >> * Dataflow Java SDK: > >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK (includes > >> the user-level API, DirectRunner, and DataflowRunner) > >> * Flink runner: https://github.com/dataArtisans/flink-dataflow > >> * Spark runner: https://github.com/cloudera/spark-dataflow > >> > >> Gathering together folks that are interested in coordinating on various > >> runners is a great idea. As we get the repo structured -- we'll make > sure > >> that that's easy to do. > >> > >> Frances > >> > >>> On Wed, Feb 10, 2016 at 4:50 PM, Yang, Connie <[email protected]> wrote: > >>> > >>> Hi Taylor, > >>> > >>> Good news indeed that the upcoming Storm 1.0 release will include > sliding > >>> and tumble windowing. > >>> > >>> Subash and I would definitely interested in joining Storm running > design > >>> and implementation. Is there an ongoing design discussion that we can > be > >>> part of? Is there a development branch we can take a look? > >>> > >>> Thanks > >>> Connie > >>> > >>>> On 2/10/16, 2:55 PM, "P. Taylor Goetz" <[email protected]> wrote: > >>>> > >>>> Hi Connie, > >>>> > >>>> I've definitely considered it (no implementation yet, just thoughts), > >> and > >>>> would like to start coordinating such an effort. That's one of the > >>>> reasons I wanted to get involved with Beam. > >>>> > >>>> A Storm runner would likely/definitely depend on features in the > >> upcoming > >>>> 1.0 release (namely MillWheel style windowing). That release is > >> currently > >>>> stabilizing, but should be out soon. > >>>> > >>>> The question I have for the Beam community would be where/how to host > >> the > >>>> collaboration effort: Storm, Beam, or a combination? > >>>> > >>>> I'm happy to facilitate with the Storm community/PMC. > >>>> > >>>> -Taylor > >>>> > >>>>> On Feb 9, 2016, at 2:07 PM, Yang, Connie <[email protected]> wrote: > >>>>> > >>>>> Hi Beamers, > >>>>> > >>>>> We¹re considering adding an Apache Storm Runner to Apache Beam. > >>>>> > >>>>> * Has anyone in the community considered this and started working > >> on > >>>>> this? > >>>>> * If so, what¹s the current state? > >>>>> > >>>>> We would like to join the design discussion and development of it if > >>>>> it¹s in flight. > >>>>> > >>>>> Thanks > >>>>> Connie > >> >
