Forwarding an email that should have been on this mailing list: ---------- Forwarded message ---------- From: Maxime Beauchemin <[email protected]> Date: Fri, Mar 17, 2017 at 8:53 AM Subject: Re: Airflow best practices To: Shreyas Joshi <[email protected]>
Hi Shreyas, Simple Airflow scripts are simply "configuration as code" and probably don't need to be abstracted out. The DSL is pretty expressive and there's usually a way to write your script so that it mostly contains code specific to your pipeline (without much boilerplate). For more advanced pipelines and complicated patterns, say dynamically building pipelines, it makes a lot of sense to create abstractions (modules, functions, classes, ...). Airflow isn't opinionated as to how you use its primitives, the only [perhaps odd] requirement is that your DAG objects should be in global module scope, somewhere in your DAGS_FOLDER, so that they can be discovered by Airflow's "DAG crawler". At Airbnb we have a lot of abstractions that generate Airflow objects. Some examples of that include our AB testing framework, common data quality enforcement patterns (stage the data, run DQ checks, exchange the partition to production), and pretty much every other Airflow script. People create the logic they need to create their pipeline, a lot of it is "as dynamic as it needs to be". It's pretty common for people to write their own operators as well, packaged with their modules. We should put some more complex examples out somewhere to show people the kinds of things that can be done, though usually programmers using Airflow realize quickly the kinds of things they can do, I'm sure you did already! Max On Fri, Mar 17, 2017 at 6:46 AM, Shreyas Joshi [email protected] <[email protected]>> wrote: > Hello Maxime, > > I am a data engineer at Github and we have been using Airflow for the last > few months. I noticed that in many of the example DAGs the code is simply > at the module level with no functions etc. Is this a recommended pattern > with Airflow DAGs? If so- I’d be very curious to know what the rationale > behind this recommendation is. > > Thanks, > Shreyas
