We might be an edge case running Airflow as a platform at Astronomer, but we make hooks and operators that are reused across many Airflow instances by customers. (Starting to open source at https://github.com/airflow-plugins.) We've also run a Mesos executor as a plugin for similar reasons so that as we fix bugs or add features we can reuse it across Airflow installs.
To add one more point — we've tossed around the idea of building more tooling around plugins, so you could do something like: $ airflow plugin install -U github-plugin then import GitHubHook and... A package install via PyPI could work as well but we haven't seen anyone else distributing Airflow plugins as packages yet. T *Taylor Edmiston* TEdmiston.com <https://www.tedmiston.com/> | Blog <http://blog.tedmiston.com> Stack Overflow CV <https://stackoverflow.com/story/taylor> | LinkedIn <https://www.linkedin.com/in/tedmiston/> | AngelList <https://angel.co/taylor> On Fri, Mar 30, 2018 at 4:37 PM, Maxime Beauchemin < [email protected]> wrote: > Yes, it makes most sense to just have a `common/hooks` and > `common/operators/` in the repo where your DAGs live and import them at > will. > > Max > > On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin <[email protected]> wrote: > > > Thanks for the responses! I think my conclusion was similar, they seem > good > > for redistribution, but if you're only working with operators and hooks > and > > aren't sharing that code then it might not make too much sense to use > them. > > > > On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin < > > [email protected]> wrote: > > > > > The original intent was to use plugins as a way to share sets of > objects > > > and applications build on top of Airflow. > > > > > > For instance it'd be possible to ship the things listed bellow as > Airflow > > > plugin: > > > > > > * "validate-and-schedule my query" UI > > > * a set of ML-related hooks and operators that match a specific > workflow > > > * a Hive metastore-browser UI > > > * drag and drop UI > > > * ... > > > > > > As far as I know it's not a super popular feature. Maybe the scope of > > > Airflow is already large enough without having all that stuff sitting > on > > > top of it. :) > > > > > > As George pointed out it could also allow to accelerate the release > > cadence > > > of sets of things that are currently inside Airflow. Things like Google > > > Cloud-related operators and hooks could ship as a plugin on their own > > > release schedule. > > > > > > Max > > > > > > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220 > > > <(949)%20412-7220> < > > > [email protected]> wrote: > > > > > > > At Netflix we've put our plugin inside the DAGs folder and pointed > the > > > > config to it there so we can both import directly in DAGs AND update > > the > > > > plugin as we go. This makes it easy to test changes to operators > needed > > > for > > > > ongoing DAG development in the same PR. > > > > > > > > The two plugin features I've used which don't translate to the direct > > > > import approach are custom macros (we provide some internal > libraries) > > > and > > > > UI menu links, which we use for linking local docs describing our > > > > deployment and custom operators, server/worker monitoring with atlas, > > and > > > > genie job monitoring. > > > > > > > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman > > > > <[email protected]> wrote: > > > > > > > > > It's presumably useful if you want to package your plugins for > other > > > > people > > > > > to use but it seems like everyone just adds those directly to the > > > Airflow > > > > > codebase these days. > > > > > > > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <[email protected]> > > > wrote: > > > > > > > > > > > Yeah so far I have only written hooks and operators so maybe the > > > > benefit > > > > > > only kicks in for other airflow abstractions. > > > > > > > > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > We also import our operators and sensors directly. > > > > > > > > > > > > > > However, executors and some other pieces are a little bit > harder > > to > > > > > deal > > > > > > > with as non-plugins > > > > > > > > > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin < > > [email protected]> > > > > > > wrote: > > > > > > >> > > > > > > >> Hello, > > > > > > >> > > > > > > >> I just got done writing a few plugins, and the process has > left > > me > > > > > > >> wondering what the real benefits are? As far as I can tell, it > > > makes > > > > > > >> testing more difficult since you cannot import from the > created > > > > > module, > > > > > > you > > > > > > >> have to import directly from the plugin. Additionally, your > code > > > > > editor > > > > > > >> isn't aware of these new plugin modules since they are created > > > when > > > > > you > > > > > > >> start the app up, this makes it seem like there are errors > when > > > > there > > > > > > >> aren't. Why not just create a lib/ dir with hooks, operators > > etc.. > > > > > dirs > > > > > > >> inside and be done with it? Very curious what peoples thoughts > > > are, > > > > > who > > > > > > >> knows I could be testing wrong or writing the plugins wrong. > > > Thanks > > > > in > > > > > > >> advance! > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
