Hey!
I've read your post and I would point that you could instead of creating
tasks on one dag for each lob, you could create one dag per each lob with a
'my_bigquery_dag_for_{{lob}}'. This way you get a more isolated unit of
management (dag). The advantage here being that in the event of a failure
or data reprocess for a single lob you wouldn't need to reprocess all lobs.
Other than that it is really cool that you're putting it out on a blog
post, it will definitely be useful.
Cheers !
On Wed, Mar 22, 2017 at 4:20 PM Andrew Maguire <[email protected]>
wrote:
> Hi All,
>
> Just to follow up as fyi.
>
> Think i figured out at least one approach. Did a blog post on it
>
> http://engineering.pmc.com/2017/03/playing-around-with-apache-airflow-bigquery-62/
>
> Can't wait to get going on porting stuff over from cron.
>
> Cheers,
> Andy
>
> On Tue, Mar 21, 2017 at 2:49 PM Andrew Maguire <[email protected]>
> wrote:
>
> > Sorry if this is not the right place for this.
> >
> > I'm wondering if anyone might be able to help me with this question?
> >
> >
> >
> http://stackoverflow.com/questions/42930221/best-way-to-loop-through-parameters-in-airflow
> >
> > Feel free to ignore if this is not the right place to try find help.
> >
> > Loving airflow btw!
> >
> > Cheers,
> > Andy
> >
>
