Hi Andy, I am a Data Eng at Airbnb and created the DAG dumper that we use internally. I have a working importable module that has been working great for us. You simply pass it the DAG object along with a few optional fields and it renders & dumps everything to file. We also include the git branch in our dumping path so we can easily do diff comparisons across our branches.
I have been looking into adding most of the features to the airflow cli but I haven't had time to do that yet (I may be able to do it during the upcoming Airflow Hackathon). I would be happy to go over our module and share it with the community. Let me know if that would be helpful, Amit > > > ---------- Forwarded message ---------- > From: Andy Welch <[email protected]> > To: [email protected] > Cc: > Date: Tue, 21 Feb 2017 15:17:06 -0800 > Subject: Dumping DAGs > Aloha! > > I'm working on a PR that will allow for dumping the configuration of a DAG > to a json record, such that my team can run analytics on the resulting > structure. I'm modifying cli.py's list_tasks command, adding an option to > request such. I thought that it's best to dump from Airflow's internal > model rather than trying to emulate how it (allows for) dynamic dag > creation. > > Please advise if there is history around this or major objections, > otherwise, I'll be sending along a PR as per the Contributor's Guide. > > Thanks for making Airflow open source :) > > -- > *Andy Welch* > [email protected] > <http://www.nerdwallet.com/?trk=emaiL_sig> > <https://www.facebook.com/NerdWallet> > <https://www.linkedin.com/company/2029368> > <https://twitter.com/NerdWallet> (415) > 549-8913 > 901 Market Street > San Francisco, CA 94103 > > Providing clarity for all of life's financial decisions > > > ---------- Forwarded message ---------- > From: Bolke de Bruin <[email protected]> > To: [email protected] > Cc: > Date: Wed, 22 Feb 2017 06:54:19 +0100 > Subject: Re: Dumping DAGs > Please make sure to code it as an API endpoint (see trigger_dag as an > example). Just changing the cli won't be enough to get the pr accepted. > > Yes that would ask you to refactor list_tasks. > > Bolke > > Sent from my iPhone > > > On 22 Feb 2017, at 00:17, Andy Welch <[email protected]> wrote: > > > > Aloha! > > > > I'm working on a PR that will allow for dumping the configuration of a > DAG > > to a json record, such that my team can run analytics on the resulting > > structure. I'm modifying cli.py's list_tasks command, adding an option > to > > request such. I thought that it's best to dump from Airflow's internal > > model rather than trying to emulate how it (allows for) dynamic dag > > creation. > > > > Please advise if there is history around this or major objections, > > otherwise, I'll be sending along a PR as per the Contributor's Guide. > > > > Thanks for making Airflow open source :) > > > > -- > > *Andy Welch* > > [email protected] > > <http://www.nerdwallet.com/?trk=emaiL_sig> > > <https://www.facebook.com/NerdWallet> > > <https://www.linkedin.com/company/2029368> > > <https://twitter.com/NerdWallet> (415) > > 549-8913 > > 901 Market Street > > San Francisco, CA 94103 > > > > Providing clarity for all of life's financial decisions > > >
