I've mailed about this before and this is a great opportunity for another shameless plug.
I consider airflow much more than a scheduler... there's a philosophy behind it that users should understand, then it makes using airflow much more effective. This site is already reachable from the wiki and shows an end-to-end example how one could do ETL work: https://gtoonstra.github.io/etl-with-airflow/ If anyone wants to contribute, let me know. If someone finds a better place for it instead of my personal github, I'm very much willing to put it in a more central location somewhere. It would also be great to receive comments on the current material, you can't really write a true " best practices" document if there's no one around to establish consensus with :). Rgds, Gerard On Fri, Mar 17, 2017 at 6:23 PM, siddharth anand <[email protected]> wrote: > FYI, we have some best practices in confluent and possibly in other places > as well. I'd recommend adding to that rather than relying on email. Email > can be used to mail the link :-) > -s > > On Fri, Mar 17, 2017 at 9:33 AM, Maxime Beauchemin < > [email protected]> wrote: > > > Forwarding an email that should have been on this mailing list: > > > > ---------- Forwarded message ---------- > > From: Maxime Beauchemin <[email protected]> > > Date: Fri, Mar 17, 2017 at 8:53 AM > > Subject: Re: Airflow best practices > > To: Shreyas Joshi <[email protected]> > > > > > > Hi Shreyas, > > > > Simple Airflow scripts are simply "configuration as code" and probably > > don't need to be abstracted out. The DSL is pretty expressive and there's > > usually a way to write your script so that it mostly contains code > specific > > to your pipeline (without much boilerplate). > > > > For more advanced pipelines and complicated patterns, say dynamically > > building pipelines, it makes a lot of sense to create abstractions > > (modules, functions, classes, ...). Airflow isn't opinionated as to how > you > > use its primitives, the only [perhaps odd] requirement is that your DAG > > objects should be in global module scope, somewhere in your DAGS_FOLDER, > so > > that they can be discovered by Airflow's "DAG crawler". > > > > At Airbnb we have a lot of abstractions that generate Airflow objects. > Some > > examples of that include our AB testing framework, common data quality > > enforcement patterns (stage the data, run DQ checks, exchange the > partition > > to production), and pretty much every other Airflow script. People create > > the logic they need to create their pipeline, a lot of it is "as dynamic > as > > it needs to be". It's pretty common for people to write their own > operators > > as well, packaged with their modules. > > > > We should put some more complex examples out somewhere to show people the > > kinds of things that can be done, though usually programmers using > Airflow > > realize quickly the kinds of things they can do, I'm sure you did > already! > > > > Max > > > > On Fri, Mar 17, 2017 at 6:46 AM, Shreyas Joshi [email protected] > > <[email protected]>> wrote: > > > > > Hello Maxime, > > > > > > I am a data engineer at Github and we have been using Airflow for the > > last > > > few months. I noticed that in many of the example DAGs the code is > simply > > > at the module level with no functions etc. Is this a recommended > pattern > > > with Airflow DAGs? If so- I’d be very curious to know what the > rationale > > > behind this recommendation is. > > > > > > Thanks, > > > Shreyas > > >
