Thanks for the responses! I think my conclusion was similar, they seem good for redistribution, but if you're only working with operators and hooks and aren't sharing that code then it might not make too much sense to use them.
On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin < [email protected]> wrote: > The original intent was to use plugins as a way to share sets of objects > and applications build on top of Airflow. > > For instance it'd be possible to ship the things listed bellow as Airflow > plugin: > > * "validate-and-schedule my query" UI > * a set of ML-related hooks and operators that match a specific workflow > * a Hive metastore-browser UI > * drag and drop UI > * ... > > As far as I know it's not a super popular feature. Maybe the scope of > Airflow is already large enough without having all that stuff sitting on > top of it. :) > > As George pointed out it could also allow to accelerate the release cadence > of sets of things that are currently inside Airflow. Things like Google > Cloud-related operators and hooks could ship as a plugin on their own > release schedule. > > Max > > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220 > <(949)%20412-7220> < > [email protected]> wrote: > > > At Netflix we've put our plugin inside the DAGs folder and pointed the > > config to it there so we can both import directly in DAGs AND update the > > plugin as we go. This makes it easy to test changes to operators needed > for > > ongoing DAG development in the same PR. > > > > The two plugin features I've used which don't translate to the direct > > import approach are custom macros (we provide some internal libraries) > and > > UI menu links, which we use for linking local docs describing our > > deployment and custom operators, server/worker monitoring with atlas, and > > genie job monitoring. > > > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman > > <[email protected]> wrote: > > > > > It's presumably useful if you want to package your plugins for other > > people > > > to use but it seems like everyone just adds those directly to the > Airflow > > > codebase these days. > > > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <[email protected]> > wrote: > > > > > > > Yeah so far I have only written hooks and operators so maybe the > > benefit > > > > only kicks in for other airflow abstractions. > > > > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman < > > > > [email protected]> wrote: > > > > > > > > > > We also import our operators and sensors directly. > > > > > > > > > > However, executors and some other pieces are a little bit harder to > > > deal > > > > > with as non-plugins > > > > > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <[email protected]> > > > > wrote: > > > > >> > > > > >> Hello, > > > > >> > > > > >> I just got done writing a few plugins, and the process has left me > > > > >> wondering what the real benefits are? As far as I can tell, it > makes > > > > >> testing more difficult since you cannot import from the created > > > module, > > > > you > > > > >> have to import directly from the plugin. Additionally, your code > > > editor > > > > >> isn't aware of these new plugin modules since they are created > when > > > you > > > > >> start the app up, this makes it seem like there are errors when > > there > > > > >> aren't. Why not just create a lib/ dir with hooks, operators etc.. > > > dirs > > > > >> inside and be done with it? Very curious what peoples thoughts > are, > > > who > > > > >> knows I could be testing wrong or writing the plugins wrong. > Thanks > > in > > > > >> advance! > > > > >> > > > > > > > > > >
