Re: What are the advantages of plugins, not sure I see any?

[Taylor Edmiston]

We might be an edge case running Airflow as a platform at Astronomer, but
we make hooks and operators that are reused across many Airflow instances
by customers.  (Starting to open source at
https://github.com/airflow-plugins.)  We've also run a Mesos executor as a
plugin for similar reasons so that as we fix bugs or add features we can
reuse it across Airflow installs.

To add one more point — we've tossed around the idea of building more
tooling around plugins, so you could do something like: $ airflow plugin
install -U github-plugin then import GitHubHook and...

A package install via PyPI could work as well but we haven't seen anyone
else distributing Airflow plugins as packages yet.

T

*Taylor Edmiston*
TEdmiston.com  | Blog

Stack Overflow CV  | LinkedIn
 | AngelList



On Fri, Mar 30, 2018 at 4:37 PM, Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> Yes, it makes most sense to just have a `common/hooks` and
> `common/operators/` in the repo where your DAGs live and import them at
> will.
>
> Max
>
> On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin  wrote:
>
> > Thanks for the responses! I think my conclusion was similar, they seem
> good
> > for redistribution, but if you're only working with operators and hooks
> and
> > aren't sharing that code then it might not make too much sense to use
> them.
> >
> > On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
> > maximebeauche...@gmail.com> wrote:
> >
> > > The original intent was to use plugins as a way to share sets of
> objects
> > > and applications build on top of Airflow.
> > >
> > > For instance it'd be possible to ship the things listed bellow as
> Airflow
> > > plugin:
> > >
> > > * "validate-and-schedule my query" UI
> > > * a set of ML-related hooks and operators that match a specific
> workflow
> > > * a Hive metastore-browser UI
> > > * drag and drop UI
> > > * ...
> > >
> > > As far as I know it's not a super popular feature. Maybe the scope of
> > > Airflow is already large enough without having all that stuff sitting
> on
> > > top of it. :)
> > >
> > > As George pointed out it could also allow to accelerate the release
> > cadence
> > > of sets of things that are currently inside Airflow. Things like Google
> > > Cloud-related operators and hooks could ship as a plugin on their own
> > > release schedule.
> > >
> > > Max
> > >
> > > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> > > <(949)%20412-7220> <
> > > alex.n.ja...@gmail.com> wrote:
> > >
> > > > At Netflix we've put our plugin inside the DAGs folder and pointed
> the
> > > > config to it there so we can both import directly in DAGs AND update
> > the
> > > > plugin as we go. This makes it easy to test changes to operators
> needed
> > > for
> > > > ongoing DAG development in the same PR.
> > > >
> > > > The two plugin features I've used which don't translate to the direct
> > > > import approach are custom macros (we provide some internal
> libraries)
> > > and
> > > > UI menu links, which we use for linking local docs describing our
> > > > deployment and custom operators, server/worker monitoring with atlas,
> > and
> > > > genie job monitoring.
> > > >
> > > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > > >  wrote:
> > > >
> > > > > It's presumably useful if you want to package your plugins for
> other
> > > > people
> > > > > to use but it seems like everyone just adds those directly to the
> > > Airflow
> > > > > codebase these days.
> > > > >
> > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin 
> > > wrote:
> > > > >
> > > > > > Yeah so far I have only written hooks and operators so maybe the
> > > > benefit
> > > > > > only  kicks in for other airflow abstractions.
> > > > > >
> > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > > > geo...@cloverhealth.com.INVALID> wrote:
> > > > > > >
> > > > > > > We also import our operators and sensors directly.
> > > > > > >
> > > > > > > However, executors and some other pieces are a little bit
> harder
> > to
> > > > > deal
> > > > > > > with as non-plugins
> > > > > > >
> > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <
> > hamlin...@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Hello,
> > > > > > >>
> > > > > > >> I just got done writing a few plugins, and the process has
> left
> > me
> > > > > > >> wondering what the real benefits are? As far as I can tell, it
> > > makes
> > > > > > >> testing more difficult since you cannot import from the
> created
> > > > > module,
> > > > > > you
> > > > > > >> have to import directly from the plugin. Additionally, your
> code
> > > > > editor
> > > > > > >> isn't aware of these new plugin modules since they are created
> > > when
> > > > > you
> > > > > > >> start 

Re: What are the advantages of plugins, not sure I see any?

[Maxime Beauchemin]

Yes, it makes most sense to just have a `common/hooks` and
`common/operators/` in the repo where your DAGs live and import them at
will.

Max

On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin  wrote:

> Thanks for the responses! I think my conclusion was similar, they seem good
> for redistribution, but if you're only working with operators and hooks and
> aren't sharing that code then it might not make too much sense to use them.
>
> On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
> > The original intent was to use plugins as a way to share sets of objects
> > and applications build on top of Airflow.
> >
> > For instance it'd be possible to ship the things listed bellow as Airflow
> > plugin:
> >
> > * "validate-and-schedule my query" UI
> > * a set of ML-related hooks and operators that match a specific workflow
> > * a Hive metastore-browser UI
> > * drag and drop UI
> > * ...
> >
> > As far as I know it's not a super popular feature. Maybe the scope of
> > Airflow is already large enough without having all that stuff sitting on
> > top of it. :)
> >
> > As George pointed out it could also allow to accelerate the release
> cadence
> > of sets of things that are currently inside Airflow. Things like Google
> > Cloud-related operators and hooks could ship as a plugin on their own
> > release schedule.
> >
> > Max
> >
> > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> > <(949)%20412-7220> <
> > alex.n.ja...@gmail.com> wrote:
> >
> > > At Netflix we've put our plugin inside the DAGs folder and pointed the
> > > config to it there so we can both import directly in DAGs AND update
> the
> > > plugin as we go. This makes it easy to test changes to operators needed
> > for
> > > ongoing DAG development in the same PR.
> > >
> > > The two plugin features I've used which don't translate to the direct
> > > import approach are custom macros (we provide some internal libraries)
> > and
> > > UI menu links, which we use for linking local docs describing our
> > > deployment and custom operators, server/worker monitoring with atlas,
> and
> > > genie job monitoring.
> > >
> > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > >  wrote:
> > >
> > > > It's presumably useful if you want to package your plugins for other
> > > people
> > > > to use but it seems like everyone just adds those directly to the
> > Airflow
> > > > codebase these days.
> > > >
> > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin 
> > wrote:
> > > >
> > > > > Yeah so far I have only written hooks and operators so maybe the
> > > benefit
> > > > > only  kicks in for other airflow abstractions.
> > > > >
> > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > > geo...@cloverhealth.com.INVALID> wrote:
> > > > > >
> > > > > > We also import our operators and sensors directly.
> > > > > >
> > > > > > However, executors and some other pieces are a little bit harder
> to
> > > > deal
> > > > > > with as non-plugins
> > > > > >
> > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <
> hamlin...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Hello,
> > > > > >>
> > > > > >> I just got done writing a few plugins, and the process has left
> me
> > > > > >> wondering what the real benefits are? As far as I can tell, it
> > makes
> > > > > >> testing more difficult since you cannot import from the created
> > > > module,
> > > > > you
> > > > > >> have to import directly from the plugin. Additionally, your code
> > > > editor
> > > > > >> isn't aware of these new plugin modules since they are created
> > when
> > > > you
> > > > > >> start the app up, this makes it seem like there are errors when
> > > there
> > > > > >> aren't. Why not just create a lib/ dir with hooks, operators
> etc..
> > > > dirs
> > > > > >> inside and be done with it? Very curious what peoples thoughts
> > are,
> > > > who
> > > > > >> knows I could be testing wrong or writing the plugins wrong.
> > Thanks
> > > in
> > > > > >> advance!
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: What are the advantages of plugins, not sure I see any?

[Kyle Hamlin]

Thanks for the responses! I think my conclusion was similar, they seem good
for redistribution, but if you're only working with operators and hooks and
aren't sharing that code then it might not make too much sense to use them.

On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> The original intent was to use plugins as a way to share sets of objects
> and applications build on top of Airflow.
>
> For instance it'd be possible to ship the things listed bellow as Airflow
> plugin:
>
> * "validate-and-schedule my query" UI
> * a set of ML-related hooks and operators that match a specific workflow
> * a Hive metastore-browser UI
> * drag and drop UI
> * ...
>
> As far as I know it's not a super popular feature. Maybe the scope of
> Airflow is already large enough without having all that stuff sitting on
> top of it. :)
>
> As George pointed out it could also allow to accelerate the release cadence
> of sets of things that are currently inside Airflow. Things like Google
> Cloud-related operators and hooks could ship as a plugin on their own
> release schedule.
>
> Max
>
> On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> <(949)%20412-7220> <
> alex.n.ja...@gmail.com> wrote:
>
> > At Netflix we've put our plugin inside the DAGs folder and pointed the
> > config to it there so we can both import directly in DAGs AND update the
> > plugin as we go. This makes it easy to test changes to operators needed
> for
> > ongoing DAG development in the same PR.
> >
> > The two plugin features I've used which don't translate to the direct
> > import approach are custom macros (we provide some internal libraries)
> and
> > UI menu links, which we use for linking local docs describing our
> > deployment and custom operators, server/worker monitoring with atlas, and
> > genie job monitoring.
> >
> > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> >  wrote:
> >
> > > It's presumably useful if you want to package your plugins for other
> > people
> > > to use but it seems like everyone just adds those directly to the
> Airflow
> > > codebase these days.
> > >
> > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin 
> wrote:
> > >
> > > > Yeah so far I have only written hooks and operators so maybe the
> > benefit
> > > > only  kicks in for other airflow abstractions.
> > > >
> > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > geo...@cloverhealth.com.INVALID> wrote:
> > > > >
> > > > > We also import our operators and sensors directly.
> > > > >
> > > > > However, executors and some other pieces are a little bit harder to
> > > deal
> > > > > with as non-plugins
> > > > >
> > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin 
> > > > wrote:
> > > > >>
> > > > >> Hello,
> > > > >>
> > > > >> I just got done writing a few plugins, and the process has left me
> > > > >> wondering what the real benefits are? As far as I can tell, it
> makes
> > > > >> testing more difficult since you cannot import from the created
> > > module,
> > > > you
> > > > >> have to import directly from the plugin. Additionally, your code
> > > editor
> > > > >> isn't aware of these new plugin modules since they are created
> when
> > > you
> > > > >> start the app up, this makes it seem like there are errors when
> > there
> > > > >> aren't. Why not just create a lib/ dir with hooks, operators etc..
> > > dirs
> > > > >> inside and be done with it? Very curious what peoples thoughts
> are,
> > > who
> > > > >> knows I could be testing wrong or writing the plugins wrong.
> Thanks
> > in
> > > > >> advance!
> > > > >>
> > > >
> > >
> >
>

Re: What are the advantages of plugins, not sure I see any?

[Maxime Beauchemin]

The original intent was to use plugins as a way to share sets of objects
and applications build on top of Airflow.

For instance it'd be possible to ship the things listed bellow as Airflow
plugin:

* "validate-and-schedule my query" UI
* a set of ML-related hooks and operators that match a specific workflow
* a Hive metastore-browser UI
* drag and drop UI
* ...

As far as I know it's not a super popular feature. Maybe the scope of
Airflow is already large enough without having all that stuff sitting on
top of it. :)

As George pointed out it could also allow to accelerate the release cadence
of sets of things that are currently inside Airflow. Things like Google
Cloud-related operators and hooks could ship as a plugin on their own
release schedule.

Max

On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220 <
alex.n.ja...@gmail.com> wrote:

> At Netflix we've put our plugin inside the DAGs folder and pointed the
> config to it there so we can both import directly in DAGs AND update the
> plugin as we go. This makes it easy to test changes to operators needed for
> ongoing DAG development in the same PR.
>
> The two plugin features I've used which don't translate to the direct
> import approach are custom macros (we provide some internal libraries) and
> UI menu links, which we use for linking local docs describing our
> deployment and custom operators, server/worker monitoring with atlas, and
> genie job monitoring.
>
> On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
>  wrote:
>
> > It's presumably useful if you want to package your plugins for other
> people
> > to use but it seems like everyone just adds those directly to the Airflow
> > codebase these days.
> >
> > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin  wrote:
> >
> > > Yeah so far I have only written hooks and operators so maybe the
> benefit
> > > only  kicks in for other airflow abstractions.
> > >
> > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > geo...@cloverhealth.com.INVALID> wrote:
> > > >
> > > > We also import our operators and sensors directly.
> > > >
> > > > However, executors and some other pieces are a little bit harder to
> > deal
> > > > with as non-plugins
> > > >
> > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin 
> > > wrote:
> > > >>
> > > >> Hello,
> > > >>
> > > >> I just got done writing a few plugins, and the process has left me
> > > >> wondering what the real benefits are? As far as I can tell, it makes
> > > >> testing more difficult since you cannot import from the created
> > module,
> > > you
> > > >> have to import directly from the plugin. Additionally, your code
> > editor
> > > >> isn't aware of these new plugin modules since they are created when
> > you
> > > >> start the app up, this makes it seem like there are errors when
> there
> > > >> aren't. Why not just create a lib/ dir with hooks, operators etc..
> > dirs
> > > >> inside and be done with it? Very curious what peoples thoughts are,
> > who
> > > >> knows I could be testing wrong or writing the plugins wrong. Thanks
> in
> > > >> advance!
> > > >>
> > >
> >
>

Re: What are the advantages of plugins, not sure I see any?

[Alex Tronchin-James 949-412-7220]

At Netflix we've put our plugin inside the DAGs folder and pointed the
config to it there so we can both import directly in DAGs AND update the
plugin as we go. This makes it easy to test changes to operators needed for
ongoing DAG development in the same PR.

The two plugin features I've used which don't translate to the direct
import approach are custom macros (we provide some internal libraries) and
UI menu links, which we use for linking local docs describing our
deployment and custom operators, server/worker monitoring with atlas, and
genie job monitoring.

On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
 wrote:

> It's presumably useful if you want to package your plugins for other people
> to use but it seems like everyone just adds those directly to the Airflow
> codebase these days.
>
> On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin  wrote:
>
> > Yeah so far I have only written hooks and operators so maybe the benefit
> > only  kicks in for other airflow abstractions.
> >
> > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > geo...@cloverhealth.com.INVALID> wrote:
> > >
> > > We also import our operators and sensors directly.
> > >
> > > However, executors and some other pieces are a little bit harder to
> deal
> > > with as non-plugins
> > >
> > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin 
> > wrote:
> > >>
> > >> Hello,
> > >>
> > >> I just got done writing a few plugins, and the process has left me
> > >> wondering what the real benefits are? As far as I can tell, it makes
> > >> testing more difficult since you cannot import from the created
> module,
> > you
> > >> have to import directly from the plugin. Additionally, your code
> editor
> > >> isn't aware of these new plugin modules since they are created when
> you
> > >> start the app up, this makes it seem like there are errors when there
> > >> aren't. Why not just create a lib/ dir with hooks, operators etc..
> dirs
> > >> inside and be done with it? Very curious what peoples thoughts are,
> who
> > >> knows I could be testing wrong or writing the plugins wrong. Thanks in
> > >> advance!
> > >>
> >
>

Re: What are the advantages of plugins, not sure I see any?

[George Leslie-Waksman]

It's presumably useful if you want to package your plugins for other people
to use but it seems like everyone just adds those directly to the Airflow
codebase these days.

On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin  wrote:

> Yeah so far I have only written hooks and operators so maybe the benefit
> only  kicks in for other airflow abstractions.
>
> > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> geo...@cloverhealth.com.INVALID> wrote:
> >
> > We also import our operators and sensors directly.
> >
> > However, executors and some other pieces are a little bit harder to deal
> > with as non-plugins
> >
> >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin 
> wrote:
> >>
> >> Hello,
> >>
> >> I just got done writing a few plugins, and the process has left me
> >> wondering what the real benefits are? As far as I can tell, it makes
> >> testing more difficult since you cannot import from the created module,
> you
> >> have to import directly from the plugin. Additionally, your code editor
> >> isn't aware of these new plugin modules since they are created when you
> >> start the app up, this makes it seem like there are errors when there
> >> aren't. Why not just create a lib/ dir with hooks, operators etc.. dirs
> >> inside and be done with it? Very curious what peoples thoughts are, who
> >> knows I could be testing wrong or writing the plugins wrong. Thanks in
> >> advance!
> >>
>

Re: What are the advantages of plugins, not sure I see any?

[Kyle Hamlin]

Yeah so far I have only written hooks and operators so maybe the benefit only  
kicks in for other airflow abstractions.

> On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman 
>  wrote:
> 
> We also import our operators and sensors directly.
> 
> However, executors and some other pieces are a little bit harder to deal
> with as non-plugins
> 
>> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin  wrote:
>> 
>> Hello,
>> 
>> I just got done writing a few plugins, and the process has left me
>> wondering what the real benefits are? As far as I can tell, it makes
>> testing more difficult since you cannot import from the created module, you
>> have to import directly from the plugin. Additionally, your code editor
>> isn't aware of these new plugin modules since they are created when you
>> start the app up, this makes it seem like there are errors when there
>> aren't. Why not just create a lib/ dir with hooks, operators etc.. dirs
>> inside and be done with it? Very curious what peoples thoughts are, who
>> knows I could be testing wrong or writing the plugins wrong. Thanks in
>> advance!
>> 

Re: What are the advantages of plugins, not sure I see any?

[George Leslie-Waksman]

We also import our operators and sensors directly.

However, executors and some other pieces are a little bit harder to deal
with as non-plugins

On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin  wrote:

> Hello,
>
> I just got done writing a few plugins, and the process has left me
> wondering what the real benefits are? As far as I can tell, it makes
> testing more difficult since you cannot import from the created module, you
> have to import directly from the plugin. Additionally, your code editor
> isn't aware of these new plugin modules since they are created when you
> start the app up, this makes it seem like there are errors when there
> aren't. Why not just create a lib/ dir with hooks, operators etc.. dirs
> inside and be done with it? Very curious what peoples thoughts are, who
> knows I could be testing wrong or writing the plugins wrong. Thanks in
> advance!
>