+1, this is a step in the right direction. On Wed, May 18, 2016 at 10:02 PM Frances Perry <[email protected]> wrote:
> Hi Beamers -- > > I’m thrilled by the recent energy and activity on writing new Beam runners! > But that also means it’s probably time for us to figure out how, as a > community, we want to support this process. ;-) > > Back near the beginning, we had a thread [1] discussing that feature > branches are the preferred way of doing development of features or > components that may take a while to reach maturity. I think new components > like runners and SDKs meet the bar to be started from a feature branch. > (Other features, like an IO connector or library of PTransforms, might also > qualify depending on complexity.) > > We should also lay out what it takes to be considered mature enough to be > merged into master, since once that happens the component gets released to > users and failing tests become blocking issues. Here are some initial > thoughts to kick off the discussion... > > In order to be merged into master, new components / major features should: > > - > > have at least 2 contributors interested in maintaining it, and 1 > committer interested in supporting it > - > > provide both end-user and developer-facing documentation > - > > have at least a basic level of unit test coverage > - > > run all existing applicable integration tests with other Beam components > and create additional tests as appropriate > > > In addition... > > A runner should: > > - > > be able to handle a subset of the model that address a significant set > of use cases (aka. ‘traditional batch’ or ‘processing time streaming’) > - > > update the capability matrix with the current status > > > An SDK* should: > > - > > provide the ability to construct graphs with all the basic building > blocks of the model (ParDo, GroupByKey, Window, Trigger, etc) > - > > begin fleshing out the common composite transforms (Count, Join, etc) > and IO connectors (Text, Kafka, etc) > - > > have at least one runner that can execute the complete model (may be a > direct runner) > - > > provide integration tests for executing against current and future > runners > > > * A note on DSLs: I think it’s important to separate out an SDK from a > DSL, because in my mind the former is by definition equivalent to the Beam > model, while the latter may select portions of the model or change the > user-visible abstractions in order to provide a domain-specific experience. > We may want to encourage some DSLs to live separately from Beam because > they may look completely non-Beam-like to their end users. But we can > probably punt this decision until we have concrete examples to discuss. > > Another fun part of this growth is that we’ll likely grow new committers. > And given the breadth of Beam, I think it would be useful to annotate our > committers [2] page with which components folks are the most knowledgeable > about. > > Looking forward to your thoughts. > > [1] > > http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201602.mbox/%3CCAAzyFAymVNpjQgZdz2BoMknnE3H9rYRbdnUemamt9Pavw8ugsw%40mail.gmail.com%3E > > [2] http://beam.incubator.apache.org/team/ >
