Operators and hooks don't need any special plugin system - simply having them as as separate Python modules which are imported using normal python semantics is enough.
In fact now that I think about it: I want to deprecate the plugins registering hooks/operators etc and limit it to only bits which a simple python import can't manage - which I think is only anything that needs to be registered with another system, such as custom routes in the web UI. I'll draft an AIP for this soon. -ash > On 18 Sep 2018, at 00:50, George Leslie-Waksman <[email protected]> wrote: > > Given we have a plugin system, could we alternatively move away from > keeping non-core supported code outside of the core project/repo? > > It would hugely decrease the surface area of the main repository and > testing infrastructure to get most of the contrib code out to its own place. > > Further, it would decrease the committer burden of having to approve/merge > code that is not supposed to be their responsibility. > > On Mon, Sep 17, 2018 at 4:37 PM Tim Swast <[email protected]> wrote: > >>> Individual operators and hooks living in separate repositories on github >> (or possibly other Apache projects), which are then distributed by pip and >> installed as libraries seems like it would scale better. >> >> Pandas did this about a year ago, and it's seemed to have worked well. For >> example, pandas.read_gbq is a very thin wrapper around pandas_gbq.read_gbq >> (distributed as a separate package). It has made it easier for me to track >> issues corresponding to my area of expertise. >> >> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <[email protected]> wrote: >> >>>> My understanding as a contributor is that if a hook/operator is in >> core, >>> it >>>> means that a committer is willing to take personal responsibility to >>>> maintain it (or at least help maintain it), and everything else goes in >>>> contrib. >>> >>> That's not correct. All of the code is owned by the entire >>> community[1]; no one person is responsible for any of it. There's no >>> silos, fiefdoms, walled gardens, etc. If the community cannot support >>> a piece of code it should be deprecated and subsequently removed. >>> >>> Contrib sections are almost always problematic for this reason. >>> Hadoop ended up abandoning its. Because Airflow acts as a gathering >>> point for so many disparate technologies (databases, storage systems, >>> compute engines, etc.), trying to keep all of them corralled and up to >>> date will be very difficult. Individual operators and hooks living in >>> separate repositories on github (or possibly other Apache projects), >>> which are then distributed by pip and installed as libraries seems >>> like it would scale better. >>> >>> -Jakob >>> >>> [1] https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie >>> >>> On 15 September 2018 at 13:29, Jeff Payne <[email protected]> wrote: >>>> How many operators are added to contrib per month? Is it too many to >>> make the decision case by case? If so, then the above mentioned rule >> sounds >>> fairly reasonable. However, if that's the rule, shouldn't a bunch of >>> existing modules be moved from contrib to core? >>>> >>>> Get Outlook for Android<https://aka.ms/ghei36> >>>> >>>> ________________________________ >>>> From: Taylor Edmiston <[email protected]> >>>> Sent: Saturday, September 15, 2018 1:13:47 PM >>>> To: [email protected] >>>> Subject: Re: Guidelines on Contrib vs Non-contrib >>>> >>>> My understanding as a contributor is that if a hook/operator is in >> core, >>> it >>>> means that a committer is willing to take personal responsibility to >>>> maintain it (or at least help maintain it), and everything else goes in >>>> contrib. >>>> >>>> *Taylor Edmiston* >>>> Blog <https://blog.tedmiston.com/> | LinkedIn >>>> <https://www.linkedin.com/in/tedmiston/> | Stack Overflow >>>> <https://stackoverflow.com/users/149428/taylor-edmiston> | Developer >>> Story >>>> <https://stackoverflow.com/story/taylor> >>>> >>>> >>>> >>>> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <[email protected]> >> wrote: >>>> >>>>> Hi, all (mainly contributors), >>>>> >>>>> Can we decide on a common guideline on when a hook/operator should go >>> under >>>>> contrib vs core? >>>>> >>>>> Regards, >>>>> >>>>> *Kaxil Naik* >>>>> *Big Data Consultant *@ *Data Reply UK* >>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & >>> Neo4j >>>>> Developer >>>>> *Phone: *+44 (0) 74820 88992 >>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil >>>>> >>> >> -- >> * • **Tim Swast* >> * • *Software Friendliness Engineer >> * • *Google Cloud Developer Relations >> * • *Seattle, WA, USA >>
