Hi Lakshmi, Thanks for volunteering.
I think Pramod's suggestion of putting the operators into 3 buckets and Siyuan's suggestion of starting a shared Google Sheet that tracks individual operators are both good, with the exception that lib/streamquery is one unit and we probably do not need to look at individual operators under it. If we don't have any objection in the community, let's start the process. David On Tue, Jul 12, 2016 at 2:24 PM, Lakshmi Velineni <laks...@datatorrent.com> wrote: > I am interested to work on this. > > Regards, > Lakshmi prasanna > > On Tue, Jul 12, 2016 at 1:55 PM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > Why not have a shared google sheet with a list of operators and options > > that we want to do with it. > > I think it's case by case. > > But retire unused or obsolete operators is important and we should do it > > sooner rather than later. > > > > Regards, > > Siyuan > > > > On Tue, Jul 12, 2016 at 1:09 PM, Amol Kekre <a...@datatorrent.com> > wrote: > > > >> > >> My vote is to do 2&3 > >> > >> Thks > >> Amol > >> > >> > >> On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < > >> vkottapa...@directv.com> wrote: > >> > >>> +1 for deprecating the packages listed below. > >>> > >>> -----Original Message----- > >>> From: hsy...@gmail.com [mailto:hsy...@gmail.com] > >>> Sent: Tuesday, July 12, 2016 12:01 PM > >>> > >>> +1 > >>> > >>> On Tue, Jul 12, 2016 at 11:53 AM, David Yan <da...@datatorrent.com> > >>> wrote: > >>> > >>> > Hi all, > >>> > > >>> > I would like to renew the discussion of retiring operators in Malhar. > >>> > > >>> > As stated before, the reason why we would like to retire operators in > >>> > Malhar is because some of them were written a long time ago before > >>> > Apache incubation, and they do not pertain to real use cases, are not > >>> > up to par in code quality, have no potential for improvement, and > >>> > probably completely unused by anybody. > >>> > > >>> > We do not want contributors to use them as a model of their > >>> > contribution, or users to use them thinking they are of quality, and > >>> then hit a wall. > >>> > Both scenarios are not beneficial to the reputation of Apex. > >>> > > >>> > The initial 3 packages that we would like to target are *lib/algo*, > >>> > *lib/math*, and *lib/streamquery*. > >>> > >>> > > >>> > I'm adding this thread to the users list. Please speak up if you are > >>> > using any operator in these 3 packages. We would like to hear from > you. > >>> > > >>> > These are the options I can think of for retiring those operators: > >>> > > >>> > 1) Completely remove them from the malhar repository. > >>> > 2) Move them from malhar-library into a separate artifact called > >>> > malhar-misc > >>> > 3) Mark them deprecated and add to their javadoc that they are no > >>> > longer supported > >>> > > >>> > Note that 2 and 3 are not mutually exclusive. Any thoughts? > >>> > > >>> > David > >>> > > >>> > On Tue, Jun 7, 2016 at 2:27 PM, Pramod Immaneni > >>> > <pra...@datatorrent.com> > >>> > wrote: > >>> > > >>> >> I wanted to close the loop on this discussion. In general everyone > >>> >> seemed to be favorable to this idea with no serious objections. > Folks > >>> >> had good suggestions like documenting capabilities of operators, > come > >>> >> up well defined criteria for graduation of operators and what those > >>> >> criteria may be and what to do with existing operators that may not > >>> >> yet be mature or unused. > >>> >> > >>> >> I am going to summarize the key points that resulted from the > >>> >> discussion and would like to proceed with them. > >>> >> > >>> >> - Operators that do not yet provide the key platform capabilities > >>> to > >>> >> make an operator useful across different applications such as > >>> >> reusability, > >>> >> partitioning static or dynamic, idempotency, exactly once will > >>> still be > >>> >> accepted as long as they are functionally correct, have unit > tests > >>> >> and will > >>> >> go into a separate module. > >>> >> - Contrib module was suggested as a place where new contributions > >>> go in > >>> >> that don't yet have all the platform capabilities and are not yet > >>> >> mature. > >>> >> If there are no other suggestions we will go with this one. > >>> >> - It was suggested the operators documentation list those > platform > >>> >> capabilities it currently provides from the list above. I will > >>> >> document a > >>> >> structure for this in the contribution guidelines. > >>> >> - Folks wanted to know what would be the criteria to graduate an > >>> >> operator to the big leagues :). I will kick-off a separate thread > >>> >> for it as > >>> >> I think it requires its own discussion and hopefully we can come > >>> >> up with a > >>> >> set of guidelines for it. > >>> >> - David brought up state of some of the existing operators and > >>> their > >>> >> retirement and the layout of operators in Malhar in general and > >>> how it > >>> >> causes problems with development. I will ask him to lead the > >>> >> discussion on > >>> >> that. > >>> >> > >>> >> Thanks > >>> >> > >>> >> On Fri, May 27, 2016 at 7:47 PM, David Yan <da...@datatorrent.com> > >>> wrote: > >>> >> > >>> >> > The two ideas are not conflicting, but rather complementing. > >>> >> > > >>> >> > On the contrary, putting a new process for people trying to > >>> >> > contribute while NOT addressing the old unused subpar operators in > >>> >> > the repository > >>> >> is > >>> >> > what is conflicting. > >>> >> > > >>> >> > Keep in mind that when people try to contribute, they always look > >>> >> > at the existing operators already in the repository as examples > and > >>> >> > likely a > >>> >> model > >>> >> > for their new operators. > >>> >> > > >>> >> > David > >>> >> > > >>> >> > > >>> >> > On Fri, May 27, 2016 at 4:05 PM, Amol Kekre <a...@datatorrent.com > > > >>> >> wrote: > >>> >> > > >>> >> > > Yes there are two conflicting threads now. The original thread > >>> >> > > was to > >>> >> > open > >>> >> > > up a way for contributors to submit code in a dir (contrib?) as > >>> >> > > long > >>> >> as > >>> >> > > license part of taken care of. > >>> >> > > > >>> >> > > On the thread of removing non-used operators -> How do we know > >>> >> > > what is being used? > >>> >> > > > >>> >> > > Thks, > >>> >> > > Amol > >>> >> > > > >>> >> > > > >>> >> > > On Fri, May 27, 2016 at 3:40 PM, Sandesh Hegde < > >>> >> sand...@datatorrent.com> > >>> >> > > wrote: > >>> >> > > > >>> >> > > > +1 for removing the not-used operators. > >>> >> > > > > >>> >> > > > So we are creating a process for operator writers who don't > >>> >> > > > want to understand the platform, yet wants to contribute? How > >>> >> > > > big is that > >>> >> set? > >>> >> > > > If we tell the app-user, here is the code which has not passed > >>> >> > > > all > >>> >> the > >>> >> > > > checklist, will they be ready to use that in production? > >>> >> > > > > >>> >> > > > This thread has 2 conflicting forces, reduce the operators and > >>> >> > > > make > >>> >> it > >>> >> > > easy > >>> >> > > > to add more operators. > >>> >> > > > > >>> >> > > > > >>> >> > > > > >>> >> > > > On Fri, May 27, 2016 at 3:03 PM Pramod Immaneni < > >>> >> > pra...@datatorrent.com> > >>> >> > > > wrote: > >>> >> > > > > >>> >> > > > > On Fri, May 27, 2016 at 2:30 PM, Gaurav Gupta < > >>> >> > > gaurav.gopi...@gmail.com> > >>> >> > > > > wrote: > >>> >> > > > > > >>> >> > > > > > Pramod, > >>> >> > > > > > > >>> >> > > > > > By that logic I would say let's put all partitionable > >>> >> > > > > > operators > >>> >> > into > >>> >> > > > one > >>> >> > > > > > folder, non-partitionable operators in another and so > on... > >>> >> > > > > > > >>> >> > > > > > >>> >> > > > > Remember the original goal of making it easier for new > >>> >> > > > > members to contribute and managing those contributions to > >>> >> > > > > maturity. It is > >>> >> not a > >>> >> > > > > functional level separation. > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > > When I look at hadoop code I see these annotations being > >>> >> > > > > > used at > >>> >> > > class > >>> >> > > > > > level and not at package/folder level. > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > I had a typo in my email, I meant to say "think of this like > >>> >> > > > > a > >>> >> > > folder..." > >>> >> > > > > as an analogy and not literally. > >>> >> > > > > > >>> >> > > > > Thanks > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > > Thanks > >>> >> > > > > > > >>> >> > > > > > On Fri, May 27, 2016 at 2:10 PM, Pramod Immaneni < > >>> >> > > > pra...@datatorrent.com > >>> >> > > > > > > >>> >> > > > > > wrote: > >>> >> > > > > > > >>> >> > > > > > > On Fri, May 27, 2016 at 1:05 PM, Gaurav Gupta < > >>> >> > > > > gaurav.gopi...@gmail.com> > >>> >> > > > > > > wrote: > >>> >> > > > > > > > >>> >> > > > > > > > Can same goal not be achieved by using > >>> >> > > org.apache.hadoop.classification.InterfaceStability.Evolving > >>> >> > > > / > >>> >> > > > > > > > > org.apache.hadoop.classification.InterfaceStability.Uns > >>> >> > > > > > > > table > >>> >> > > > > > annotation? > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > I think it is important to localize the additions in one > >>> >> place so > >>> >> > > > that > >>> >> > > > > it > >>> >> > > > > > > becomes clearer to users about the maturity level of > >>> >> > > > > > > these, > >>> >> > easier > >>> >> > > > for > >>> >> > > > > > > developers to track them towards the path to maturity > and > >>> >> > > > > > > also > >>> >> > > > > provides a > >>> >> > > > > > > clearer directive for committers and contributors on > >>> >> acceptance > >>> >> > of > >>> >> > > > new > >>> >> > > > > > > submissions. Relying on the annotations alone makes them > >>> >> spread > >>> >> > all > >>> >> > > > > over > >>> >> > > > > > > the place and adds an additional layer of difficulty in > >>> >> > > > identification > >>> >> > > > > > not > >>> >> > > > > > > just for users but also for developers who want to find > >>> >> > > > > > > such > >>> >> > > > operators > >>> >> > > > > > and > >>> >> > > > > > > improve them. This of this like a folder level > annotation > >>> >> where > >>> >> > > > > > everything > >>> >> > > > > > > under this folder is unstable or evolving. > >>> >> > > > > > > > >>> >> > > > > > > Thanks > >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > On Fri, May 27, 2016 at 12:35 PM, David Yan < > >>> >> > > da...@datatorrent.com > >>> >> > > > > > >>> >> > > > > > > wrote: > >>> >> > > > > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > > > > > > > >>> >> > > > > > > > > > > > > >>> >> > > > > > > > > > > > Malhar in its current state, has way too many > >>> >> operators > >>> >> > > > that > >>> >> > > > > > fall > >>> >> > > > > > > > in > >>> >> > > > > > > > > > the > >>> >> > > > > > > > > > > > "non-production quality" category. We should > >>> >> > > > > > > > > > > > make it > >>> >> > > > obvious > >>> >> > > > > to > >>> >> > > > > > > > users > >>> >> > > > > > > > > > > that > >>> >> > > > > > > > > > > > which operators are up to par, and which > >>> >> > > > > > > > > > > > operators > >>> >> are > >>> >> > > not, > >>> >> > > > > and > >>> >> > > > > > > > maybe > >>> >> > > > > > > > > > > even > >>> >> > > > > > > > > > > > remove those that are likely not ever used in > a > >>> >> > > > > > > > > > > > real > >>> >> > use > >>> >> > > > > case. > >>> >> > > > > > > > > > > > > >>> >> > > > > > > > > > > > >>> >> > > > > > > > > > > I am ambivalent about revisiting older operators > >>> >> > > > > > > > > > > and > >>> >> > doing > >>> >> > > > this > >>> >> > > > > > > > > exercise > >>> >> > > > > > > > > > as > >>> >> > > > > > > > > > > this can cause unnecessary tensions. My original > >>> >> intent > >>> >> > is > >>> >> > > > for > >>> >> > > > > > > > > > > contributions going forward. > >>> >> > > > > > > > > > > > >>> >> > > > > > > > > > > > >>> >> > > > > > > > > > IMO it is important to address this as well. > >>> >> > > > > > > > > > Operators > >>> >> > > outside > >>> >> > > > > the > >>> >> > > > > > > play > >>> >> > > > > > > > > > area should be of well known quality. > >>> >> > > > > > > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > > > > I think this is important, and I don't anticipate > >>> >> > > > > > > > > much > >>> >> > tension > >>> >> > > if > >>> >> > > > > we > >>> >> > > > > > > > > establish clear criteria. > >>> >> > > > > > > > > It's not helpful if we let the old subpar operators > >>> >> > > > > > > > > stay > >>> >> and > >>> >> > > put > >>> >> > > > up > >>> >> > > > > > the > >>> >> > > > > > > > > bars for new operators. > >>> >> > > > > > > > > > >>> >> > > > > > > > > David > >>> >> > > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > >>> >> > > > > > >>> >> > > > > >>> >> > > > >>> >> > > >>> >> > >>> > > >>> > > >>> > >> > >> > > >
