Yes, it makes most sense to just have a `common/hooks` and `common/operators/` in the repo where your DAGs live and import them at will.
Max On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin <[email protected]> wrote: > Thanks for the responses! I think my conclusion was similar, they seem good > for redistribution, but if you're only working with operators and hooks and > aren't sharing that code then it might not make too much sense to use them. > > On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin < > [email protected]> wrote: > > > The original intent was to use plugins as a way to share sets of objects > > and applications build on top of Airflow. > > > > For instance it'd be possible to ship the things listed bellow as Airflow > > plugin: > > > > * "validate-and-schedule my query" UI > > * a set of ML-related hooks and operators that match a specific workflow > > * a Hive metastore-browser UI > > * drag and drop UI > > * ... > > > > As far as I know it's not a super popular feature. Maybe the scope of > > Airflow is already large enough without having all that stuff sitting on > > top of it. :) > > > > As George pointed out it could also allow to accelerate the release > cadence > > of sets of things that are currently inside Airflow. Things like Google > > Cloud-related operators and hooks could ship as a plugin on their own > > release schedule. > > > > Max > > > > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220 > > <(949)%20412-7220> < > > [email protected]> wrote: > > > > > At Netflix we've put our plugin inside the DAGs folder and pointed the > > > config to it there so we can both import directly in DAGs AND update > the > > > plugin as we go. This makes it easy to test changes to operators needed > > for > > > ongoing DAG development in the same PR. > > > > > > The two plugin features I've used which don't translate to the direct > > > import approach are custom macros (we provide some internal libraries) > > and > > > UI menu links, which we use for linking local docs describing our > > > deployment and custom operators, server/worker monitoring with atlas, > and > > > genie job monitoring. > > > > > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman > > > <[email protected]> wrote: > > > > > > > It's presumably useful if you want to package your plugins for other > > > people > > > > to use but it seems like everyone just adds those directly to the > > Airflow > > > > codebase these days. > > > > > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <[email protected]> > > wrote: > > > > > > > > > Yeah so far I have only written hooks and operators so maybe the > > > benefit > > > > > only kicks in for other airflow abstractions. > > > > > > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman < > > > > > [email protected]> wrote: > > > > > > > > > > > > We also import our operators and sensors directly. > > > > > > > > > > > > However, executors and some other pieces are a little bit harder > to > > > > deal > > > > > > with as non-plugins > > > > > > > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin < > [email protected]> > > > > > wrote: > > > > > >> > > > > > >> Hello, > > > > > >> > > > > > >> I just got done writing a few plugins, and the process has left > me > > > > > >> wondering what the real benefits are? As far as I can tell, it > > makes > > > > > >> testing more difficult since you cannot import from the created > > > > module, > > > > > you > > > > > >> have to import directly from the plugin. Additionally, your code > > > > editor > > > > > >> isn't aware of these new plugin modules since they are created > > when > > > > you > > > > > >> start the app up, this makes it seem like there are errors when > > > there > > > > > >> aren't. Why not just create a lib/ dir with hooks, operators > etc.. > > > > dirs > > > > > >> inside and be done with it? Very curious what peoples thoughts > > are, > > > > who > > > > > >> knows I could be testing wrong or writing the plugins wrong. > > Thanks > > > in > > > > > >> advance! > > > > > >> > > > > > > > > > > > > > > >
