Added comments, also recommend having the misc folder for the remaining operators in contrib according to proposed guidelines
https://github.com/apache/apex-site/pull/44 On Mon, Aug 1, 2016 at 12:22 AM, Lakshmi Velineni <[email protected]> wrote: > Hi > > I also added recommendation for lib/math operators to the same document as > a separate sheet. Please have a look. > > Thanks > Lakshmi Prasanna > > On Fri, Jul 29, 2016 at 7:19 PM, Lakshmi Velineni <[email protected] > > wrote: > >> Hi, >> >> I also added recommendation for each operator . Please take a look. >> >> thanks >> >> On Thu, Jul 28, 2016 at 11:03 AM, Lakshmi Velineni < >> [email protected]> wrote: >> >>> Hi, >>> >>> I created a shared google sheet and tracked the various details of >>> operators. Currently, the sheet contains information about operators under >>> lib/algo only. Link is https://docs.google.com/a/ >>> datatorrent.com/spreadsheets/d/15IjMa-vYK6Wru4kZnUGIgg6sQAptDJ_ >>> CaWpXt3GDccM/edit?usp=sharing . Will update the sheet soon with >>> lib/math too. >>> >>> Thanks >>> Lakshmi Prasanna >>> >>> On Tue, Jul 12, 2016 at 2:36 PM, David Yan <[email protected]> >>> wrote: >>> >>>> Hi Lakshmi, >>>> >>>> Thanks for volunteering. >>>> >>>> I think Pramod's suggestion of putting the operators into 3 buckets and >>>> Siyuan's suggestion of starting a shared Google Sheet that tracks >>>> individual operators are both good, with the exception that lib/streamquery >>>> is one unit and we probably do not need to look at individual operators >>>> under it. >>>> >>>> If we don't have any objection in the community, let's start the >>>> process. >>>> >>>> David >>>> >>>> On Tue, Jul 12, 2016 at 2:24 PM, Lakshmi Velineni < >>>> [email protected]> wrote: >>>> >>>>> I am interested to work on this. >>>>> >>>>> Regards, >>>>> Lakshmi prasanna >>>>> >>>>> On Tue, Jul 12, 2016 at 1:55 PM, [email protected] <[email protected]> >>>>> wrote: >>>>> >>>>> > Why not have a shared google sheet with a list of operators and >>>>> options >>>>> > that we want to do with it. >>>>> > I think it's case by case. >>>>> > But retire unused or obsolete operators is important and we should >>>>> do it >>>>> > sooner rather than later. >>>>> > >>>>> > Regards, >>>>> > Siyuan >>>>> > >>>>> > On Tue, Jul 12, 2016 at 1:09 PM, Amol Kekre <[email protected]> >>>>> wrote: >>>>> > >>>>> >> >>>>> >> My vote is to do 2&3 >>>>> >> >>>>> >> Thks >>>>> >> Amol >>>>> >> >>>>> >> >>>>> >> On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < >>>>> >> [email protected]> wrote: >>>>> >> >>>>> >>> +1 for deprecating the packages listed below. >>>>> >>> >>>>> >>> -----Original Message----- >>>>> >>> From: [email protected] [mailto:[email protected]] >>>>> >>> Sent: Tuesday, July 12, 2016 12:01 PM >>>>> >>> >>>>> >>> +1 >>>>> >>> >>>>> >>> On Tue, Jul 12, 2016 at 11:53 AM, David Yan <[email protected] >>>>> > >>>>> >>> wrote: >>>>> >>> >>>>> >>> > Hi all, >>>>> >>> > >>>>> >>> > I would like to renew the discussion of retiring operators in >>>>> Malhar. >>>>> >>> > >>>>> >>> > As stated before, the reason why we would like to retire >>>>> operators in >>>>> >>> > Malhar is because some of them were written a long time ago >>>>> before >>>>> >>> > Apache incubation, and they do not pertain to real use cases, >>>>> are not >>>>> >>> > up to par in code quality, have no potential for improvement, and >>>>> >>> > probably completely unused by anybody. >>>>> >>> > >>>>> >>> > We do not want contributors to use them as a model of their >>>>> >>> > contribution, or users to use them thinking they are of quality, >>>>> and >>>>> >>> then hit a wall. >>>>> >>> > Both scenarios are not beneficial to the reputation of Apex. >>>>> >>> > >>>>> >>> > The initial 3 packages that we would like to target are >>>>> *lib/algo*, >>>>> >>> > *lib/math*, and *lib/streamquery*. >>>>> >>> >>>>> >>> > >>>>> >>> > I'm adding this thread to the users list. Please speak up if you >>>>> are >>>>> >>> > using any operator in these 3 packages. We would like to hear >>>>> from you. >>>>> >>> > >>>>> >>> > These are the options I can think of for retiring those >>>>> operators: >>>>> >>> > >>>>> >>> > 1) Completely remove them from the malhar repository. >>>>> >>> > 2) Move them from malhar-library into a separate artifact called >>>>> >>> > malhar-misc >>>>> >>> > 3) Mark them deprecated and add to their javadoc that they are no >>>>> >>> > longer supported >>>>> >>> > >>>>> >>> > Note that 2 and 3 are not mutually exclusive. Any thoughts? >>>>> >>> > >>>>> >>> > David >>>>> >>> > >>>>> >>> > On Tue, Jun 7, 2016 at 2:27 PM, Pramod Immaneni >>>>> >>> > <[email protected]> >>>>> >>> > wrote: >>>>> >>> > >>>>> >>> >> I wanted to close the loop on this discussion. In general >>>>> everyone >>>>> >>> >> seemed to be favorable to this idea with no serious objections. >>>>> Folks >>>>> >>> >> had good suggestions like documenting capabilities of >>>>> operators, come >>>>> >>> >> up well defined criteria for graduation of operators and what >>>>> those >>>>> >>> >> criteria may be and what to do with existing operators that may >>>>> not >>>>> >>> >> yet be mature or unused. >>>>> >>> >> >>>>> >>> >> I am going to summarize the key points that resulted from the >>>>> >>> >> discussion and would like to proceed with them. >>>>> >>> >> >>>>> >>> >> - Operators that do not yet provide the key platform >>>>> capabilities >>>>> >>> to >>>>> >>> >> make an operator useful across different applications such as >>>>> >>> >> reusability, >>>>> >>> >> partitioning static or dynamic, idempotency, exactly once >>>>> will >>>>> >>> still be >>>>> >>> >> accepted as long as they are functionally correct, have unit >>>>> tests >>>>> >>> >> and will >>>>> >>> >> go into a separate module. >>>>> >>> >> - Contrib module was suggested as a place where new >>>>> contributions >>>>> >>> go in >>>>> >>> >> that don't yet have all the platform capabilities and are >>>>> not yet >>>>> >>> >> mature. >>>>> >>> >> If there are no other suggestions we will go with this one. >>>>> >>> >> - It was suggested the operators documentation list those >>>>> platform >>>>> >>> >> capabilities it currently provides from the list above. I >>>>> will >>>>> >>> >> document a >>>>> >>> >> structure for this in the contribution guidelines. >>>>> >>> >> - Folks wanted to know what would be the criteria to >>>>> graduate an >>>>> >>> >> operator to the big leagues :). I will kick-off a separate >>>>> thread >>>>> >>> >> for it as >>>>> >>> >> I think it requires its own discussion and hopefully we can >>>>> come >>>>> >>> >> up with a >>>>> >>> >> set of guidelines for it. >>>>> >>> >> - David brought up state of some of the existing operators >>>>> and >>>>> >>> their >>>>> >>> >> retirement and the layout of operators in Malhar in general >>>>> and >>>>> >>> how it >>>>> >>> >> causes problems with development. I will ask him to lead the >>>>> >>> >> discussion on >>>>> >>> >> that. >>>>> >>> >> >>>>> >>> >> Thanks >>>>> >>> >> >>>>> >>> >> On Fri, May 27, 2016 at 7:47 PM, David Yan < >>>>> [email protected]> >>>>> >>> wrote: >>>>> >>> >> >>>>> >>> >> > The two ideas are not conflicting, but rather complementing. >>>>> >>> >> > >>>>> >>> >> > On the contrary, putting a new process for people trying to >>>>> >>> >> > contribute while NOT addressing the old unused subpar >>>>> operators in >>>>> >>> >> > the repository >>>>> >>> >> is >>>>> >>> >> > what is conflicting. >>>>> >>> >> > >>>>> >>> >> > Keep in mind that when people try to contribute, they always >>>>> look >>>>> >>> >> > at the existing operators already in the repository as >>>>> examples and >>>>> >>> >> > likely a >>>>> >>> >> model >>>>> >>> >> > for their new operators. >>>>> >>> >> > >>>>> >>> >> > David >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > On Fri, May 27, 2016 at 4:05 PM, Amol Kekre < >>>>> [email protected]> >>>>> >>> >> wrote: >>>>> >>> >> > >>>>> >>> >> > > Yes there are two conflicting threads now. The original >>>>> thread >>>>> >>> >> > > was to >>>>> >>> >> > open >>>>> >>> >> > > up a way for contributors to submit code in a dir >>>>> (contrib?) as >>>>> >>> >> > > long >>>>> >>> >> as >>>>> >>> >> > > license part of taken care of. >>>>> >>> >> > > >>>>> >>> >> > > On the thread of removing non-used operators -> How do we >>>>> know >>>>> >>> >> > > what is being used? >>>>> >>> >> > > >>>>> >>> >> > > Thks, >>>>> >>> >> > > Amol >>>>> >>> >> > > >>>>> >>> >> > > >>>>> >>> >> > > On Fri, May 27, 2016 at 3:40 PM, Sandesh Hegde < >>>>> >>> >> [email protected]> >>>>> >>> >> > > wrote: >>>>> >>> >> > > >>>>> >>> >> > > > +1 for removing the not-used operators. >>>>> >>> >> > > > >>>>> >>> >> > > > So we are creating a process for operator writers who >>>>> don't >>>>> >>> >> > > > want to understand the platform, yet wants to contribute? >>>>> How >>>>> >>> >> > > > big is that >>>>> >>> >> set? >>>>> >>> >> > > > If we tell the app-user, here is the code which has not >>>>> passed >>>>> >>> >> > > > all >>>>> >>> >> the >>>>> >>> >> > > > checklist, will they be ready to use that in production? >>>>> >>> >> > > > >>>>> >>> >> > > > This thread has 2 conflicting forces, reduce the >>>>> operators and >>>>> >>> >> > > > make >>>>> >>> >> it >>>>> >>> >> > > easy >>>>> >>> >> > > > to add more operators. >>>>> >>> >> > > > >>>>> >>> >> > > > >>>>> >>> >> > > > >>>>> >>> >> > > > On Fri, May 27, 2016 at 3:03 PM Pramod Immaneni < >>>>> >>> >> > [email protected]> >>>>> >>> >> > > > wrote: >>>>> >>> >> > > > >>>>> >>> >> > > > > On Fri, May 27, 2016 at 2:30 PM, Gaurav Gupta < >>>>> >>> >> > > [email protected]> >>>>> >>> >> > > > > wrote: >>>>> >>> >> > > > > >>>>> >>> >> > > > > > Pramod, >>>>> >>> >> > > > > > >>>>> >>> >> > > > > > By that logic I would say let's put all partitionable >>>>> >>> >> > > > > > operators >>>>> >>> >> > into >>>>> >>> >> > > > one >>>>> >>> >> > > > > > folder, non-partitionable operators in another and so >>>>> on... >>>>> >>> >> > > > > > >>>>> >>> >> > > > > >>>>> >>> >> > > > > Remember the original goal of making it easier for new >>>>> >>> >> > > > > members to contribute and managing those contributions >>>>> to >>>>> >>> >> > > > > maturity. It is >>>>> >>> >> not a >>>>> >>> >> > > > > functional level separation. >>>>> >>> >> > > > > >>>>> >>> >> > > > > >>>>> >>> >> > > > > > When I look at hadoop code I see these annotations >>>>> being >>>>> >>> >> > > > > > used at >>>>> >>> >> > > class >>>>> >>> >> > > > > > level and not at package/folder level. >>>>> >>> >> > > > > >>>>> >>> >> > > > > >>>>> >>> >> > > > > I had a typo in my email, I meant to say "think of this >>>>> like >>>>> >>> >> > > > > a >>>>> >>> >> > > folder..." >>>>> >>> >> > > > > as an analogy and not literally. >>>>> >>> >> > > > > >>>>> >>> >> > > > > Thanks >>>>> >>> >> > > > > >>>>> >>> >> > > > > >>>>> >>> >> > > > > > Thanks >>>>> >>> >> > > > > > >>>>> >>> >> > > > > > On Fri, May 27, 2016 at 2:10 PM, Pramod Immaneni < >>>>> >>> >> > > > [email protected] >>>>> >>> >> > > > > > >>>>> >>> >> > > > > > wrote: >>>>> >>> >> > > > > > >>>>> >>> >> > > > > > > On Fri, May 27, 2016 at 1:05 PM, Gaurav Gupta < >>>>> >>> >> > > > > [email protected]> >>>>> >>> >> > > > > > > wrote: >>>>> >>> >> > > > > > > >>>>> >>> >> > > > > > > > Can same goal not be achieved by using >>>>> >>> >> > > org.apache.hadoop.classification. >>>>> InterfaceStability.Evolving >>>>> >>> >> > > > / >>>>> >>> >> > > > > > > > org.apache.hadoop.classification. >>>>> InterfaceStability.Uns >>>>> >>> >> > > > > > > > table >>>>> >>> >> > > > > > annotation? >>>>> >>> >> > > > > > > > >>>>> >>> >> > > > > > > >>>>> >>> >> > > > > > > I think it is important to localize the additions >>>>> in one >>>>> >>> >> place so >>>>> >>> >> > > > that >>>>> >>> >> > > > > it >>>>> >>> >> > > > > > > becomes clearer to users about the maturity level of >>>>> >>> >> > > > > > > these, >>>>> >>> >> > easier >>>>> >>> >> > > > for >>>>> >>> >> > > > > > > developers to track them towards the path to >>>>> maturity and >>>>> >>> >> > > > > > > also >>>>> >>> >> > > > > provides a >>>>> >>> >> > > > > > > clearer directive for committers and contributors on >>>>> >>> >> acceptance >>>>> >>> >> > of >>>>> >>> >> > > > new >>>>> >>> >> > > > > > > submissions. Relying on the annotations alone makes >>>>> them >>>>> >>> >> spread >>>>> >>> >> > all >>>>> >>> >> > > > > over >>>>> >>> >> > > > > > > the place and adds an additional layer of >>>>> difficulty in >>>>> >>> >> > > > identification >>>>> >>> >> > > > > > not >>>>> >>> >> > > > > > > just for users but also for developers who want to >>>>> find >>>>> >>> >> > > > > > > such >>>>> >>> >> > > > operators >>>>> >>> >> > > > > > and >>>>> >>> >> > > > > > > improve them. This of this like a folder level >>>>> annotation >>>>> >>> >> where >>>>> >>> >> > > > > > everything >>>>> >>> >> > > > > > > under this folder is unstable or evolving. >>>>> >>> >> > > > > > > >>>>> >>> >> > > > > > > Thanks >>>>> >>> >> > > > > > > >>>>> >>> >> > > > > > > >>>>> >>> >> > > > > > > > >>>>> >>> >> > > > > > > > On Fri, May 27, 2016 at 12:35 PM, David Yan < >>>>> >>> >> > > [email protected] >>>>> >>> >> > > > > >>>>> >>> >> > > > > > > wrote: >>>>> >>> >> > > > > > > > >>>>> >>> >> > > > > > > > > > >>>>> >>> >> > > > > > > > > > > >>>>> >>> >> > > > > > > > > > > > >>>>> >>> >> > > > > > > > > > > > Malhar in its current state, has way too >>>>> many >>>>> >>> >> operators >>>>> >>> >> > > > that >>>>> >>> >> > > > > > fall >>>>> >>> >> > > > > > > > in >>>>> >>> >> > > > > > > > > > the >>>>> >>> >> > > > > > > > > > > > "non-production quality" category. We >>>>> should >>>>> >>> >> > > > > > > > > > > > make it >>>>> >>> >> > > > obvious >>>>> >>> >> > > > > to >>>>> >>> >> > > > > > > > users >>>>> >>> >> > > > > > > > > > > that >>>>> >>> >> > > > > > > > > > > > which operators are up to par, and which >>>>> >>> >> > > > > > > > > > > > operators >>>>> >>> >> are >>>>> >>> >> > > not, >>>>> >>> >> > > > > and >>>>> >>> >> > > > > > > > maybe >>>>> >>> >> > > > > > > > > > > even >>>>> >>> >> > > > > > > > > > > > remove those that are likely not ever >>>>> used in a >>>>> >>> >> > > > > > > > > > > > real >>>>> >>> >> > use >>>>> >>> >> > > > > case. >>>>> >>> >> > > > > > > > > > > > >>>>> >>> >> > > > > > > > > > > >>>>> >>> >> > > > > > > > > > > I am ambivalent about revisiting older >>>>> operators >>>>> >>> >> > > > > > > > > > > and >>>>> >>> >> > doing >>>>> >>> >> > > > this >>>>> >>> >> > > > > > > > > exercise >>>>> >>> >> > > > > > > > > > as >>>>> >>> >> > > > > > > > > > > this can cause unnecessary tensions. My >>>>> original >>>>> >>> >> intent >>>>> >>> >> > is >>>>> >>> >> > > > for >>>>> >>> >> > > > > > > > > > > contributions going forward. >>>>> >>> >> > > > > > > > > > > >>>>> >>> >> > > > > > > > > > > >>>>> >>> >> > > > > > > > > > IMO it is important to address this as well. >>>>> >>> >> > > > > > > > > > Operators >>>>> >>> >> > > outside >>>>> >>> >> > > > > the >>>>> >>> >> > > > > > > play >>>>> >>> >> > > > > > > > > > area should be of well known quality. >>>>> >>> >> > > > > > > > > > >>>>> >>> >> > > > > > > > > > >>>>> >>> >> > > > > > > > > I think this is important, and I don't >>>>> anticipate >>>>> >>> >> > > > > > > > > much >>>>> >>> >> > tension >>>>> >>> >> > > if >>>>> >>> >> > > > > we >>>>> >>> >> > > > > > > > > establish clear criteria. >>>>> >>> >> > > > > > > > > It's not helpful if we let the old subpar >>>>> operators >>>>> >>> >> > > > > > > > > stay >>>>> >>> >> and >>>>> >>> >> > > put >>>>> >>> >> > > > up >>>>> >>> >> > > > > > the >>>>> >>> >> > > > > > > > > bars for new operators. >>>>> >>> >> > > > > > > > > >>>>> >>> >> > > > > > > > > David >>>>> >>> >> > > > > > > > > >>>>> >>> >> > > > > > > > >>>>> >>> >> > > > > > > >>>>> >>> >> > > > > > >>>>> >>> >> > > > > >>>>> >>> >> > > > >>>>> >>> >> > > >>>>> >>> >> > >>>>> >>> >> >>>>> >>> > >>>>> >>> > >>>>> >>> >>>>> >> >>>>> >> >>>>> > >>>>> >>>> >>>> >>> >> >
