On Fri, May 27, 2016 at 10:59 AM, Priyanka Gugale <[email protected]> wrote:
> I like the idea. I am just worried about left with unreadable code as time > passes. As few people already pointed out, I would suggest there should be > kind of checklist which they should mark as done/not-done. Not done is > fine, only thing is better if developer call that out. Also if he is adding > any new parameters or any complicated logic, let's make sure it's well > documented. > Checklist and a set of guidelines for pro-actively managing the code in this space have been suggested and supported by many and should be part of the plan if the proposal goes forward. These should ensure that things don't get out of hand. Thanks > > -Priyanka > > On Fri, May 27, 2016 at 10:52 AM, Pramod Immaneni <[email protected]> > wrote: > > > On Fri, May 27, 2016 at 10:29 AM, Chinmay Kolhatkar < > > [email protected] > > > wrote: > > > > > +1 for the idea of incubating space in malhar. This will not just > ensure > > > more contributions but a faster development of operators for putting > into > > > firsthand use. > > > > > > Though, I believe we need to build a process around that, right from > the > > > point where operator developer starts to develop till the operator > > matures > > > and goes into some stable space in malhar. But the process should be > > > lightweight enough for it not to become a new reason for lesser > > > contributions. > > > > > > > Agreed, we should call out the steps and have well documented guidelines > > for these. As Thomas pointed out earlier, contributions guidelines > document > > is a good place to put these. > > > > > > > > > > Here are some thoughts around process (contributing guidelines), I > think > > we > > > need to take care of for this: > > > 1. What class address spaces the operators space should go in. This is > to > > > ensure that there are lesser backward compatibility issues exist after > > > moving to stable space. In the best scenario, it should be the change > of > > > maven dependency and everything else should work fine. > > > > 2. As Ilya pointed out, min checklist criteria is required for both > getting > > > into incubating space & then into stable space. This will ensure > nothing > > is > > > missed and the quality is maintained. > > > 3. IMO its very important to decide when, what, why and who of moving > the > > > code from incubating to stable space. This can be a grey area and > process > > > should define that upfront. Last thing we want to do in have some code > in > > > incubating but not moved to stable space for ages. > > > 4. We should also decide about the cases where some serious changes are > > > required in stable code, should we move it back to incubating and go > > > through the process again? > > > 5. Maybe we can go for Commit-Then-Review model for this incubating > > space. > > > Its important to keep in mind that we're enabling faster development at > > the > > > cost of probably more work in longer run. Hence suggesting CTR model > > > instead of RTC model. > > > 6. We should also make it clear that though this is an incubating > space, > > > there should not be any lax on unit testing of the operators. > > > > > > > I agree with most of what you have said and some of these should be > > included in the guidelines. I would say we need further discussions on > the > > actual mechanisms for 3. and 4. and these could start immediately after > the > > current proposal, if acceptable to the group, is put in place. Regd 5, I > > think we should stick with the RTC model for now and evaluate how it is > > working before considering the alternative. > > > > Thanks > > > > > > > Thanks, > > > Chinmay. > > > > > > > > > On Fri, May 27, 2016 at 10:24 AM, Pramod Immaneni < > > [email protected]> > > > wrote: > > > > > > > My comment inline > > > > > > > > On Fri, May 27, 2016 at 9:00 AM, David Yan <[email protected]> > > > wrote: > > > > > > > > > This is an important change because not only will it help > > contributors > > > > who > > > > > want to contribute to Apex Malhar, it also helps users on deciding > > > which > > > > > operators they can use. > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > Malhar in its current state, has way too many operators that fall > in > > > the > > > > > "non-production quality" category. We should make it obvious to > users > > > > that > > > > > which operators are up to par, and which operators are not, and > maybe > > > > even > > > > > remove those that are likely not ever used in a real use case. > > > > > > > > > > > > > I am ambivalent about revisiting older operators and doing this > > exercise > > > as > > > > this can cause unnecessary tensions. My original intent is for > > > > contributions going forward. > > > > > > > > Thanks > > > > > > > > > > > > > David > > > > > > > > > > On Fri, May 27, 2016 at 7:13 AM, Amol Kekre <[email protected]> > > > > wrote: > > > > > > > > > > > This is a very good idea and will greatly help contributors. The > > > > > > requirements to submit code to this Malhar folder should be very > > > > > minimal. A > > > > > > few that come to my mind > > > > > > > > > > > > - Should compile > > > > > > - License of the external lib (if any) should be Apache compliant > > > > license > > > > > > // Need to see if this is part of ASF guidelines > > > > > > > > > > > > Everything else including naming, idempotency, performance, ... > > > should > > > > be > > > > > > waived. > > > > > > > > > > > > Thks > > > > > > Amol > > > > > > > > > > > > > > > > > > On Thu, May 26, 2016 at 11:25 PM, Pramod Immaneni < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > As you all know the continued success and growth of an open > > source > > > > > > project > > > > > > > is dependent on new members joining the project and > contributing. > > > > This > > > > > > will > > > > > > > depend on how accessible the project is for new folks to make > > > > > meaningful > > > > > > > contributions. For a project like ours where the code base has > > been > > > > in > > > > > > > development for years, it can be quite daunting for new members > > to > > > > just > > > > > > > pick up and make contributions. We need to find ways to make it > > > > easier > > > > > > for > > > > > > > people to do so. Malhar, namely the operator library, is an > area > > > > where > > > > > > > people can contribute without requiring deep knowledge or > > > expertise. > > > > > > > > > > > > > > We have seen operators take time to mature as evidenced by the > > road > > > > > taken > > > > > > > by some of our commonly used operators to reach production > > quality. > > > > > This > > > > > > is > > > > > > > due to the fact that apart from the core functionality the > > operator > > > > is > > > > > > > trying to implement there are many other aspects to address > such > > as > > > > > > > performance, idempotency, processing semantics and scalability. > > It > > > > > would > > > > > > be > > > > > > > difficult even for folks familiar with all these aspects to get > > > > > > everything > > > > > > > right the first time around and produce comprehensive operators > > let > > > > > alone > > > > > > > first time contributors. At the same time operators cannot > reach > > > this > > > > > > > maturity level unless they get used in different scenarios and > > get > > > a > > > > > good > > > > > > > look at by different people. In maturity I am also including > API > > > > > > stability. > > > > > > > > > > > > > > I would like to propose creation of a space inside Malhar, a > > > > > sub-folder, > > > > > > > where contributions can first go in if they are not fully ready > > and > > > > > when > > > > > > > they mature can be moved out of the sub-folder into an existing > > > > module > > > > > > or a > > > > > > > new module of its own, the package paths can remain the same. > The > > > > > > > evaluation bar for contributions into this space would be more > > > > > permissive > > > > > > > than it is today, it would require the functionality the > operator > > > was > > > > > > > developed for be working but will not necessitate that all > fault > > > > > tolerant > > > > > > > and scalability aspects be addressed. It will also allow new > > > > operators > > > > > > that > > > > > > > are variations of existing operators till such time as we can > > > > determine > > > > > > if > > > > > > > the new functionality can be subsumed by the original operator > or > > > it > > > > > > makes > > > > > > > sense for the new operator to exist as a separate entity. It > will > > > be > > > > > > up-to > > > > > > > committers and contributors to work together and make the > > decisions > > > > as > > > > > to > > > > > > > whether the individual contributions go into this space or are > > > ready > > > > to > > > > > > > just go into the regular modules. > > > > > > > > > > > > > > What does everyone think. > > > > > > > > > > > > > > Thanks, > > > > > > > Pramod > > > > > > > > > > > > > > > > > > > > > > > > > > > >
