@Amit Fair enough, given we don't currently meet those requirements on master ;-) But still a good things to aim for, with lots of help and encouragement all around!
On Thu, May 19, 2016 at 10:47 PM, Jean-Baptiste Onofré <[email protected]> wrote: > Fully agree with Davor for feature idea impl. > > Regards > JB > > > On 05/19/2016 08:59 PM, Davor Bonaci wrote: > >> If anybody wants to experiment a little with a feature idea -- absolutely, >> individual forked repositories are certainly an awesome place for such >> attempts. >> >> However, for something that is a significant undertaking, like a new >> runner >> or new SDK, I think feature branches in the main repository make total >> sense. We'd avoid important disadvantages of lower visibility, harder for >> others to jump in, comment, learn, etc., harder testing because Apache >> Jenkins wouldn't be able to do it automatically, etc. >> >> In summary, I think there's a spectrum of feature complexities and >> longevity considerations. As such, I'd support being flexible as >> appropriate, but have a default answer of starting with a feature branch >> in >> the main repository for new major components. >> >> On Thu, May 19, 2016 at 3:09 AM, Ismaël Mejía <[email protected]> wrote: >> >> I agree with Aljoscha, about not putting the feature branches in the main >>> repo, however how can we make people aware of the new developments ? >>> >>> -Ismaël >>> >>> On Thu, May 19, 2016 at 11:56 AM, Aljoscha Krettek <[email protected]> >>> wrote: >>> >>> +1 >>>> >>>> When we say feature branch, are we talking about a branch in the main >>>> >>> repo? >>> >>>> I would propose that feature branches live in the repos of the >>>> committers >>>> who are working on a feature. >>>> >>>> On Thu, 19 May 2016 at 11:54 Jean-Baptiste Onofré <[email protected]> >>>> >>> wrote: >>> >>>> >>>> +1 >>>>> >>>>> it looks good to me. >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> On 05/19/2016 07:01 AM, Frances Perry wrote: >>>>> >>>>>> Hi Beamers -- >>>>>> >>>>>> I’m thrilled by the recent energy and activity on writing new Beam >>>>>> >>>>> runners! >>>>> >>>>>> But that also means it’s probably time for us to figure out how, as a >>>>>> community, we want to support this process. ;-) >>>>>> >>>>>> Back near the beginning, we had a thread [1] discussing that feature >>>>>> branches are the preferred way of doing development of features or >>>>>> components that may take a while to reach maturity. I think new >>>>>> >>>>> components >>>>> >>>>>> like runners and SDKs meet the bar to be started from a feature >>>>>> >>>>> branch. >>> >>>> (Other features, like an IO connector or library of PTransforms, >>>>>> >>>>> might >>> >>>> also >>>>> >>>>>> qualify depending on complexity.) >>>>>> >>>>>> We should also lay out what it takes to be considered mature enough >>>>>> >>>>> to >>> >>>> be >>>> >>>>> merged into master, since once that happens the component gets >>>>>> >>>>> released >>> >>>> to >>>>> >>>>>> users and failing tests become blocking issues. Here are some initial >>>>>> thoughts to kick off the discussion... >>>>>> >>>>>> In order to be merged into master, new components / major features >>>>>> >>>>> should: >>>>> >>>>>> >>>>>> - >>>>>> >>>>>> have at least 2 contributors interested in maintaining it, and 1 >>>>>> committer interested in supporting it >>>>>> - >>>>>> >>>>>> provide both end-user and developer-facing documentation >>>>>> - >>>>>> >>>>>> have at least a basic level of unit test coverage >>>>>> - >>>>>> >>>>>> run all existing applicable integration tests with other Beam >>>>>> >>>>> components >>>>> >>>>>> and create additional tests as appropriate >>>>>> >>>>>> >>>>>> In addition... >>>>>> >>>>>> A runner should: >>>>>> >>>>>> - >>>>>> >>>>>> be able to handle a subset of the model that address a >>>>>> >>>>> significant >>> >>>> set >>>>> >>>>>> of use cases (aka. ‘traditional batch’ or ‘processing time >>>>>> >>>>> streaming’) >>>>> >>>>>> - >>>>>> >>>>>> update the capability matrix with the current status >>>>>> >>>>>> >>>>>> An SDK* should: >>>>>> >>>>>> - >>>>>> >>>>>> provide the ability to construct graphs with all the basic >>>>>> >>>>> building >>> >>>> blocks of the model (ParDo, GroupByKey, Window, Trigger, etc) >>>>>> - >>>>>> >>>>>> begin fleshing out the common composite transforms (Count, Join, >>>>>> >>>>> etc) >>>> >>>>> and IO connectors (Text, Kafka, etc) >>>>>> - >>>>>> >>>>>> have at least one runner that can execute the complete model (may >>>>>> >>>>> be >>>> >>>>> a >>>>> >>>>>> direct runner) >>>>>> - >>>>>> >>>>>> provide integration tests for executing against current and >>>>>> >>>>> future >>> >>>> runners >>>>>> >>>>>> >>>>>> * A note on DSLs: I think it’s important to separate out an SDK >>>>>> >>>>> from a >>> >>>> DSL, because in my mind the former is by definition equivalent to the >>>>>> >>>>> Beam >>>>> >>>>>> model, while the latter may select portions of the model or change >>>>>> >>>>> the >>> >>>> user-visible abstractions in order to provide a domain-specific >>>>>> >>>>> experience. >>>>> >>>>>> We may want to encourage some DSLs to live separately from Beam >>>>>> >>>>> because >>> >>>> they may look completely non-Beam-like to their end users. But we can >>>>>> probably punt this decision until we have concrete examples to >>>>>> >>>>> discuss. >>> >>>> >>>>>> Another fun part of this growth is that we’ll likely grow new >>>>>> >>>>> committers. >>>> >>>>> And given the breadth of Beam, I think it would be useful to annotate >>>>>> >>>>> our >>>> >>>>> committers [2] page with which components folks are the most >>>>>> >>>>> knowledgeable >>>>> >>>>>> about. >>>>>> >>>>>> Looking forward to your thoughts. >>>>>> >>>>>> [1] >>>>>> >>>>>> >>>>> >>>> >>> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201602.mbox/%3CCAAzyFAymVNpjQgZdz2BoMknnE3H9rYRbdnUemamt9Pavw8ugsw%40mail.gmail.com%3E >>> >>>> >>>>>> [2] http://beam.incubator.apache.org/team/ >>>>>> >>>>>> >>>>> -- >>>>> Jean-Baptiste Onofré >>>>> [email protected] >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >>>>> >>>> >>> >> > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
