Thanks for the proposal to drop the barrier. This will enable a lot of functional contribution to the code base. We can always make a given operator production ready as and when an opportunity arises. I would go even further and categorize operators into the following which require various degrees of graduation criteria for maturity.
1. Input Operators --> needs partitioning, fault tolerance, idempotency 2. Output Operators --> needs partitioning, fault tolerance, exactly once writes behavior 3. Stateless Processing Operators --> default 4. Stateful Processing Operators --> needs partitioning, fault tolerance Also, we should have a common goal as a community to keep maturing operators. Regards, Ashwin. On Fri, May 27, 2016 at 9:59 AM, Sasha Parfenov <sas...@apache.org> wrote: > +1 > > I think this is a great idea to lower the barrier for entry and encourage > more contributions for Apex. And checklist for each operator is very > valuable for end-users who do not want to spend time going through every > line of code, to provide better understanding and confidence in various > operator aspects including processing semantics, partitioning, idempotency, > etc. > > Thanks, > Sasha > > On Fri, May 27, 2016 at 9:32 AM, Pramod Immaneni <pra...@datatorrent.com> > wrote: > > > I think this is a good idea and the committer can help in determining > this > > without putting all the burden on the contributor. One way would be to > list > > what is missing in terms of platform capabilities and under what > scenarios > > the operators cannot be used or unsure whether they will work. As you > > mentioned it could be informally done in a javadoc or we could introduce > > special programming language constructs to denote these. > > > > On Fri, May 27, 2016 at 8:22 AM, Ganelin, Ilya < > > ilya.gane...@capitalone.com> > > wrote: > > > > > If we're going to adopt partially completed code, I propose that every > > new > > > Malhar operator then contain a checklist as a comment within the class. > > > > > > This checklist would be defined by the community and would track the > > > current development state of the operator. That way there are no > > unexpected > > > surprises. > > > > > > > > > > > > Sent with Good (www.good.com) > > > ________________________________ > > > From: Amol Kekre <a...@datatorrent.com> > > > Sent: Friday, May 27, 2016 10:13:06 AM > > > To: dev@apex.apache.org > > > Cc: d...@apex.incubator.apache.org > > > Subject: Re: A proposal for Malhar > > > > > > This is a very good idea and will greatly help contributors. The > > > requirements to submit code to this Malhar folder should be very > > minimal. A > > > few that come to my mind > > > > > > - Should compile > > > - License of the external lib (if any) should be Apache compliant > license > > > // Need to see if this is part of ASF guidelines > > > > > > Everything else including naming, idempotency, performance, ... should > be > > > waived. > > > > > > Thks > > > Amol > > > > > > > > > On Thu, May 26, 2016 at 11:25 PM, Pramod Immaneni < > > pra...@datatorrent.com> > > > wrote: > > > > > > > As you all know the continued success and growth of an open source > > > project > > > > is dependent on new members joining the project and contributing. > This > > > will > > > > depend on how accessible the project is for new folks to make > > meaningful > > > > contributions. For a project like ours where the code base has been > in > > > > development for years, it can be quite daunting for new members to > just > > > > pick up and make contributions. We need to find ways to make it > easier > > > for > > > > people to do so. Malhar, namely the operator library, is an area > where > > > > people can contribute without requiring deep knowledge or expertise. > > > > > > > > We have seen operators take time to mature as evidenced by the road > > taken > > > > by some of our commonly used operators to reach production quality. > > This > > > is > > > > due to the fact that apart from the core functionality the operator > is > > > > trying to implement there are many other aspects to address such as > > > > performance, idempotency, processing semantics and scalability. It > > would > > > be > > > > difficult even for folks familiar with all these aspects to get > > > everything > > > > right the first time around and produce comprehensive operators let > > alone > > > > first time contributors. At the same time operators cannot reach this > > > > maturity level unless they get used in different scenarios and get a > > good > > > > look at by different people. In maturity I am also including API > > > stability. > > > > > > > > I would like to propose creation of a space inside Malhar, a > > sub-folder, > > > > where contributions can first go in if they are not fully ready and > > when > > > > they mature can be moved out of the sub-folder into an existing > module > > > or a > > > > new module of its own, the package paths can remain the same. The > > > > evaluation bar for contributions into this space would be more > > permissive > > > > than it is today, it would require the functionality the operator was > > > > developed for be working but will not necessitate that all fault > > tolerant > > > > and scalability aspects be addressed. It will also allow new > operators > > > that > > > > are variations of existing operators till such time as we can > determine > > > if > > > > the new functionality can be subsumed by the original operator or it > > > makes > > > > sense for the new operator to exist as a separate entity. It will be > > > up-to > > > > committers and contributors to work together and make the decisions > as > > to > > > > whether the individual contributions go into this space or are ready > to > > > > just go into the regular modules. > > > > > > > > What does everyone think. > > > > > > > > Thanks, > > > > Pramod > > > > > > > ________________________________________________________ > > > > > > The information contained in this e-mail is confidential and/or > > > proprietary to Capital One and/or its affiliates and may only be used > > > solely in performance of work or services for Capital One. The > > information > > > transmitted herewith is intended only for use by the individual or > entity > > > to which it is addressed. If the reader of this message is not the > > intended > > > recipient, you are hereby notified that any review, retransmission, > > > dissemination, distribution, copying or other use of, or taking of any > > > action in reliance upon this information is strictly prohibited. If you > > > have received this communication in error, please contact the sender > and > > > delete the material from your computer. > > > > > > -- Regards, Ashwin.
