On Fri, May 27, 2016 at 1:05 PM, Ashwin Chandra Putta < [email protected]> wrote:
> Thanks for the proposal to drop the barrier. This will enable a lot of > functional contribution to the code base. We can always make a given > operator production ready as and when an opportunity arises. I would go > even further and categorize operators into the following which require > various degrees of graduation criteria for maturity. > > 1. Input Operators --> needs partitioning, fault tolerance, idempotency > 2. Output Operators --> needs partitioning, fault tolerance, exactly once > writes behavior > 3. Stateless Processing Operators --> default > 4. Stateful Processing Operators --> needs partitioning, fault tolerance > > Also, we should have a common goal as a community to keep maturing > operators. > > Regards, > Ashwin. > Useful input please present them when we have deeper discussions on the maturity guidelines. Thanks > > On Fri, May 27, 2016 at 9:59 AM, Sasha Parfenov <[email protected]> wrote: > > > +1 > > > > I think this is a great idea to lower the barrier for entry and encourage > > more contributions for Apex. And checklist for each operator is very > > valuable for end-users who do not want to spend time going through every > > line of code, to provide better understanding and confidence in various > > operator aspects including processing semantics, partitioning, > idempotency, > > etc. > > > > Thanks, > > Sasha > > > > On Fri, May 27, 2016 at 9:32 AM, Pramod Immaneni <[email protected] > > > > wrote: > > > > > I think this is a good idea and the committer can help in determining > > this > > > without putting all the burden on the contributor. One way would be to > > list > > > what is missing in terms of platform capabilities and under what > > scenarios > > > the operators cannot be used or unsure whether they will work. As you > > > mentioned it could be informally done in a javadoc or we could > introduce > > > special programming language constructs to denote these. > > > > > > On Fri, May 27, 2016 at 8:22 AM, Ganelin, Ilya < > > > [email protected]> > > > wrote: > > > > > > > If we're going to adopt partially completed code, I propose that > every > > > new > > > > Malhar operator then contain a checklist as a comment within the > class. > > > > > > > > This checklist would be defined by the community and would track the > > > > current development state of the operator. That way there are no > > > unexpected > > > > surprises. > > > > > > > > > > > > > > > > Sent with Good (www.good.com) > > > > ________________________________ > > > > From: Amol Kekre <[email protected]> > > > > Sent: Friday, May 27, 2016 10:13:06 AM > > > > To: [email protected] > > > > Cc: [email protected] > > > > Subject: Re: A proposal for Malhar > > > > > > > > This is a very good idea and will greatly help contributors. The > > > > requirements to submit code to this Malhar folder should be very > > > minimal. A > > > > few that come to my mind > > > > > > > > - Should compile > > > > - License of the external lib (if any) should be Apache compliant > > license > > > > // Need to see if this is part of ASF guidelines > > > > > > > > Everything else including naming, idempotency, performance, ... > should > > be > > > > waived. > > > > > > > > Thks > > > > Amol > > > > > > > > > > > > On Thu, May 26, 2016 at 11:25 PM, Pramod Immaneni < > > > [email protected]> > > > > wrote: > > > > > > > > > As you all know the continued success and growth of an open source > > > > project > > > > > is dependent on new members joining the project and contributing. > > This > > > > will > > > > > depend on how accessible the project is for new folks to make > > > meaningful > > > > > contributions. For a project like ours where the code base has been > > in > > > > > development for years, it can be quite daunting for new members to > > just > > > > > pick up and make contributions. We need to find ways to make it > > easier > > > > for > > > > > people to do so. Malhar, namely the operator library, is an area > > where > > > > > people can contribute without requiring deep knowledge or > expertise. > > > > > > > > > > We have seen operators take time to mature as evidenced by the road > > > taken > > > > > by some of our commonly used operators to reach production quality. > > > This > > > > is > > > > > due to the fact that apart from the core functionality the operator > > is > > > > > trying to implement there are many other aspects to address such as > > > > > performance, idempotency, processing semantics and scalability. It > > > would > > > > be > > > > > difficult even for folks familiar with all these aspects to get > > > > everything > > > > > right the first time around and produce comprehensive operators let > > > alone > > > > > first time contributors. At the same time operators cannot reach > this > > > > > maturity level unless they get used in different scenarios and get > a > > > good > > > > > look at by different people. In maturity I am also including API > > > > stability. > > > > > > > > > > I would like to propose creation of a space inside Malhar, a > > > sub-folder, > > > > > where contributions can first go in if they are not fully ready and > > > when > > > > > they mature can be moved out of the sub-folder into an existing > > module > > > > or a > > > > > new module of its own, the package paths can remain the same. The > > > > > evaluation bar for contributions into this space would be more > > > permissive > > > > > than it is today, it would require the functionality the operator > was > > > > > developed for be working but will not necessitate that all fault > > > tolerant > > > > > and scalability aspects be addressed. It will also allow new > > operators > > > > that > > > > > are variations of existing operators till such time as we can > > determine > > > > if > > > > > the new functionality can be subsumed by the original operator or > it > > > > makes > > > > > sense for the new operator to exist as a separate entity. It will > be > > > > up-to > > > > > committers and contributors to work together and make the decisions > > as > > > to > > > > > whether the individual contributions go into this space or are > ready > > to > > > > > just go into the regular modules. > > > > > > > > > > What does everyone think. > > > > > > > > > > Thanks, > > > > > Pramod > > > > > > > > > ________________________________________________________ > > > > > > > > The information contained in this e-mail is confidential and/or > > > > proprietary to Capital One and/or its affiliates and may only be used > > > > solely in performance of work or services for Capital One. The > > > information > > > > transmitted herewith is intended only for use by the individual or > > entity > > > > to which it is addressed. If the reader of this message is not the > > > intended > > > > recipient, you are hereby notified that any review, retransmission, > > > > dissemination, distribution, copying or other use of, or taking of > any > > > > action in reliance upon this information is strictly prohibited. If > you > > > > have received this communication in error, please contact the sender > > and > > > > delete the material from your computer. > > > > > > > > > > > > > -- > > Regards, > Ashwin. >
