> Individual operators and hooks living in separate repositories on github (or possibly other Apache projects), which are then distributed by pip and installed as libraries seems like it would scale better.
Pandas did this about a year ago, and it's seemed to have worked well. For example, pandas.read_gbq is a very thin wrapper around pandas_gbq.read_gbq (distributed as a separate package). It has made it easier for me to track issues corresponding to my area of expertise. On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <jgho...@gmail.com> wrote: > > My understanding as a contributor is that if a hook/operator is in core, > it > > means that a committer is willing to take personal responsibility to > > maintain it (or at least help maintain it), and everything else goes in > > contrib. > > That's not correct. All of the code is owned by the entire > community[1]; no one person is responsible for any of it. There's no > silos, fiefdoms, walled gardens, etc. If the community cannot support > a piece of code it should be deprecated and subsequently removed. > > Contrib sections are almost always problematic for this reason. > Hadoop ended up abandoning its. Because Airflow acts as a gathering > point for so many disparate technologies (databases, storage systems, > compute engines, etc.), trying to keep all of them corralled and up to > date will be very difficult. Individual operators and hooks living in > separate repositories on github (or possibly other Apache projects), > which are then distributed by pip and installed as libraries seems > like it would scale better. > > -Jakob > > [1] https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie > > On 15 September 2018 at 13:29, Jeff Payne <jpa...@bombora.com> wrote: > > How many operators are added to contrib per month? Is it too many to > make the decision case by case? If so, then the above mentioned rule sounds > fairly reasonable. However, if that's the rule, shouldn't a bunch of > existing modules be moved from contrib to core? > > > > Get Outlook for Android<https://aka.ms/ghei36> > > > > ________________________________ > > From: Taylor Edmiston <tedmis...@gmail.com> > > Sent: Saturday, September 15, 2018 1:13:47 PM > > To: dev@airflow.incubator.apache.org > > Subject: Re: Guidelines on Contrib vs Non-contrib > > > > My understanding as a contributor is that if a hook/operator is in core, > it > > means that a committer is willing to take personal responsibility to > > maintain it (or at least help maintain it), and everything else goes in > > contrib. > > > > *Taylor Edmiston* > > Blog <https://blog.tedmiston.com/> | LinkedIn > > <https://www.linkedin.com/in/tedmiston/> | Stack Overflow > > <https://stackoverflow.com/users/149428/taylor-edmiston> | Developer > Story > > <https://stackoverflow.com/story/taylor> > > > > > > > > On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > >> Hi, all (mainly contributors), > >> > >> Can we decide on a common guideline on when a hook/operator should go > under > >> contrib vs core? > >> > >> Regards, > >> > >> *Kaxil Naik* > >> *Big Data Consultant *@ *Data Reply UK* > >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & > Neo4j > >> Developer > >> *Phone: *+44 (0) 74820 88992 > >> *LinkedIn*: https://www.linkedin.com/in/kaxil > >> > -- * • **Tim Swast* * • *Software Friendliness Engineer * • *Google Cloud Developer Relations * • *Seattle, WA, USA
