Yeah was thinking that afterwards. Would that be a sub dag approach, or is it even as easy as moving the dag definition itself into the for loop?
Cheers Andy On Wed, Mar 22, 2017 at 4:45 PM Jorge Alpedrinha Ramos < [email protected]> wrote: > Hey! > > I've read your post and I would point that you could instead of creating > tasks on one dag for each lob, you could create one dag per each lob with a > 'my_bigquery_dag_for_{{lob}}'. This way you get a more isolated unit of > management (dag). The advantage here being that in the event of a failure > or data reprocess for a single lob you wouldn't need to reprocess all lobs. > > Other than that it is really cool that you're putting it out on a blog > post, it will definitely be useful. > > Cheers ! > > On Wed, Mar 22, 2017 at 4:20 PM Andrew Maguire <[email protected]> > wrote: > > > Hi All, > > > > Just to follow up as fyi. > > > > Think i figured out at least one approach. Did a blog post on it > > > > > http://engineering.pmc.com/2017/03/playing-around-with-apache-airflow-bigquery-62/ > > > > Can't wait to get going on porting stuff over from cron. > > > > Cheers, > > Andy > > > > On Tue, Mar 21, 2017 at 2:49 PM Andrew Maguire <[email protected]> > > wrote: > > > > > Sorry if this is not the right place for this. > > > > > > I'm wondering if anyone might be able to help me with this question? > > > > > > > > > > > > http://stackoverflow.com/questions/42930221/best-way-to-loop-through-parameters-in-airflow > > > > > > Feel free to ignore if this is not the right place to try find help. > > > > > > Loving airflow btw! > > > > > > Cheers, > > > Andy > > > > > >
