FYI, we have some best practices in confluent and possibly in other places as well. I'd recommend adding to that rather than relying on email. Email can be used to mail the link :-) -s
On Fri, Mar 17, 2017 at 9:33 AM, Maxime Beauchemin < [email protected]> wrote: > Forwarding an email that should have been on this mailing list: > > ---------- Forwarded message ---------- > From: Maxime Beauchemin <[email protected]> > Date: Fri, Mar 17, 2017 at 8:53 AM > Subject: Re: Airflow best practices > To: Shreyas Joshi <[email protected]> > > > Hi Shreyas, > > Simple Airflow scripts are simply "configuration as code" and probably > don't need to be abstracted out. The DSL is pretty expressive and there's > usually a way to write your script so that it mostly contains code specific > to your pipeline (without much boilerplate). > > For more advanced pipelines and complicated patterns, say dynamically > building pipelines, it makes a lot of sense to create abstractions > (modules, functions, classes, ...). Airflow isn't opinionated as to how you > use its primitives, the only [perhaps odd] requirement is that your DAG > objects should be in global module scope, somewhere in your DAGS_FOLDER, so > that they can be discovered by Airflow's "DAG crawler". > > At Airbnb we have a lot of abstractions that generate Airflow objects. Some > examples of that include our AB testing framework, common data quality > enforcement patterns (stage the data, run DQ checks, exchange the partition > to production), and pretty much every other Airflow script. People create > the logic they need to create their pipeline, a lot of it is "as dynamic as > it needs to be". It's pretty common for people to write their own operators > as well, packaged with their modules. > > We should put some more complex examples out somewhere to show people the > kinds of things that can be done, though usually programmers using Airflow > realize quickly the kinds of things they can do, I'm sure you did already! > > Max > > On Fri, Mar 17, 2017 at 6:46 AM, Shreyas Joshi [email protected] > <[email protected]>> wrote: > > > Hello Maxime, > > > > I am a data engineer at Github and we have been using Airflow for the > last > > few months. I noticed that in many of the example DAGs the code is simply > > at the module level with no functions etc. Is this a recommended pattern > > with Airflow DAGs? If so- I’d be very curious to know what the rationale > > behind this recommendation is. > > > > Thanks, > > Shreyas >
