The original intent was to use plugins as a way to share sets of objects and applications build on top of Airflow.
For instance it'd be possible to ship the things listed bellow as Airflow plugin: * "validate-and-schedule my query" UI * a set of ML-related hooks and operators that match a specific workflow * a Hive metastore-browser UI * drag and drop UI * ... As far as I know it's not a super popular feature. Maybe the scope of Airflow is already large enough without having all that stuff sitting on top of it. :) As George pointed out it could also allow to accelerate the release cadence of sets of things that are currently inside Airflow. Things like Google Cloud-related operators and hooks could ship as a plugin on their own release schedule. Max On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220 < [email protected]> wrote: > At Netflix we've put our plugin inside the DAGs folder and pointed the > config to it there so we can both import directly in DAGs AND update the > plugin as we go. This makes it easy to test changes to operators needed for > ongoing DAG development in the same PR. > > The two plugin features I've used which don't translate to the direct > import approach are custom macros (we provide some internal libraries) and > UI menu links, which we use for linking local docs describing our > deployment and custom operators, server/worker monitoring with atlas, and > genie job monitoring. > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman > <[email protected]> wrote: > > > It's presumably useful if you want to package your plugins for other > people > > to use but it seems like everyone just adds those directly to the Airflow > > codebase these days. > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <[email protected]> wrote: > > > > > Yeah so far I have only written hooks and operators so maybe the > benefit > > > only kicks in for other airflow abstractions. > > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman < > > > [email protected]> wrote: > > > > > > > > We also import our operators and sensors directly. > > > > > > > > However, executors and some other pieces are a little bit harder to > > deal > > > > with as non-plugins > > > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <[email protected]> > > > wrote: > > > >> > > > >> Hello, > > > >> > > > >> I just got done writing a few plugins, and the process has left me > > > >> wondering what the real benefits are? As far as I can tell, it makes > > > >> testing more difficult since you cannot import from the created > > module, > > > you > > > >> have to import directly from the plugin. Additionally, your code > > editor > > > >> isn't aware of these new plugin modules since they are created when > > you > > > >> start the app up, this makes it seem like there are errors when > there > > > >> aren't. Why not just create a lib/ dir with hooks, operators etc.. > > dirs > > > >> inside and be done with it? Very curious what peoples thoughts are, > > who > > > >> knows I could be testing wrong or writing the plugins wrong. Thanks > in > > > >> advance! > > > >> > > > > > >
