yes, please!
> On Sep 18, 2018, at 12:23 PM, Maxime Beauchemin <[email protected]>
> wrote:
>
> +1 for deprecating operators/hooks as plugins, let's use Python's good old
> python packages and maybe python "entry points" if we want to inject them
> in "airflow.operators"/"airflow.hooks" (which is probably not necessary)
>
> On Tue, Sep 18, 2018 at 2:12 AM Ash Berlin-Taylor <[email protected]> wrote:
>
>> Operators and hooks don't need any special plugin system - simply having
>> them as as separate Python modules which are imported using normal python
>> semantics is enough.
>>
>> In fact now that I think about it: I want to deprecate the plugins
>> registering hooks/operators etc and limit it to only bits which a simple
>> python import can't manage - which I think is only anything that needs to
>> be registered with another system, such as custom routes in the web UI.
>>
>> I'll draft an AIP for this soon.
>>
>> -ash
>>
>>
>>> On 18 Sep 2018, at 00:50, George Leslie-Waksman <[email protected]>
>> wrote:
>>>
>>> Given we have a plugin system, could we alternatively move away from
>>> keeping non-core supported code outside of the core project/repo?
>>>
>>> It would hugely decrease the surface area of the main repository and
>>> testing infrastructure to get most of the contrib code out to its own
>> place.
>>>
>>> Further, it would decrease the committer burden of having to
>> approve/merge
>>> code that is not supposed to be their responsibility.
>>>
>>> On Mon, Sep 17, 2018 at 4:37 PM Tim Swast <[email protected]>
>> wrote:
>>>
>>>>> Individual operators and hooks living in separate repositories on
>> github
>>>> (or possibly other Apache projects), which are then distributed by pip
>> and
>>>> installed as libraries seems like it would scale better.
>>>>
>>>> Pandas did this about a year ago, and it's seemed to have worked well.
>> For
>>>> example, pandas.read_gbq is a very thin wrapper around
>> pandas_gbq.read_gbq
>>>> (distributed as a separate package). It has made it easier for me to
>> track
>>>> issues corresponding to my area of expertise.
>>>>
>>>> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <[email protected]> wrote:
>>>>
>>>>>> My understanding as a contributor is that if a hook/operator is in
>>>> core,
>>>>> it
>>>>>> means that a committer is willing to take personal responsibility to
>>>>>> maintain it (or at least help maintain it), and everything else goes
>> in
>>>>>> contrib.
>>>>>
>>>>> That's not correct. All of the code is owned by the entire
>>>>> community[1]; no one person is responsible for any of it. There's no
>>>>> silos, fiefdoms, walled gardens, etc. If the community cannot support
>>>>> a piece of code it should be deprecated and subsequently removed.
>>>>>
>>>>> Contrib sections are almost always problematic for this reason.
>>>>> Hadoop ended up abandoning its. Because Airflow acts as a gathering
>>>>> point for so many disparate technologies (databases, storage systems,
>>>>> compute engines, etc.), trying to keep all of them corralled and up to
>>>>> date will be very difficult. Individual operators and hooks living in
>>>>> separate repositories on github (or possibly other Apache projects),
>>>>> which are then distributed by pip and installed as libraries seems
>>>>> like it would scale better.
>>>>>
>>>>> -Jakob
>>>>>
>>>>> [1]
>> https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie
>>>>>
>>>>> On 15 September 2018 at 13:29, Jeff Payne <[email protected]> wrote:
>>>>>> How many operators are added to contrib per month? Is it too many to
>>>>> make the decision case by case? If so, then the above mentioned rule
>>>> sounds
>>>>> fairly reasonable. However, if that's the rule, shouldn't a bunch of
>>>>> existing modules be moved from contrib to core?
>>>>>>
>>>>>> Get Outlook for Android<https://aka.ms/ghei36>
>>>>>>
>>>>>> ________________________________
>>>>>> From: Taylor Edmiston <[email protected]>
>>>>>> Sent: Saturday, September 15, 2018 1:13:47 PM
>>>>>> To: [email protected]
>>>>>> Subject: Re: Guidelines on Contrib vs Non-contrib
>>>>>>
>>>>>> My understanding as a contributor is that if a hook/operator is in
>>>> core,
>>>>> it
>>>>>> means that a committer is willing to take personal responsibility to
>>>>>> maintain it (or at least help maintain it), and everything else goes
>> in
>>>>>> contrib.
>>>>>>
>>>>>> *Taylor Edmiston*
>>>>>> Blog <https://blog.tedmiston.com/> | LinkedIn
>>>>>> <https://www.linkedin.com/in/tedmiston/> | Stack Overflow
>>>>>> <https://stackoverflow.com/users/149428/taylor-edmiston> | Developer
>>>>> Story
>>>>>> <https://stackoverflow.com/story/taylor>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <[email protected]>
>>>> wrote:
>>>>>>
>>>>>>> Hi, all (mainly contributors),
>>>>>>>
>>>>>>> Can we decide on a common guideline on when a hook/operator should go
>>>>> under
>>>>>>> contrib vs core?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> *Kaxil Naik*
>>>>>>> *Big Data Consultant *@ *Data Reply UK*
>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>>>>> Neo4j
>>>>>>> Developer
>>>>>>> *Phone: *+44 (0) 74820 88992
>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>
>>>>>
>>>> --
>>>> * • **Tim Swast*
>>>> * • *Software Friendliness Engineer
>>>> * • *Google Cloud Developer Relations
>>>> * • *Seattle, WA, USA
>>>>
>>
>>
