An alternative approach would be to use the ExternalTaskSensor to sense when the task(s)/data that you depend upon has finished/arrived.
https://pythonhosted.org/airflow/code.html#airflow.operators.ExternalTaskSensor On Mon, Jun 13, 2016 at 9:08 AM, Laura Lorenz <[email protected]> wrote: > We do something like that here here in this way, and I'd love feedback from > anyone else if they do it a different way. Basically we use > TriggerDagRunOperators and a task in the "Producer" that cares what day of > the week it is being run, and a "Consumer" DAG with no schedule interval > that depends on being triggers when the right conditions of the "Producer" > DAG are met. > > In more detail, we have a daily DAG like your "Producer" that ingests data > every day. We want to run a series of tasks including generating a report > in a "Consumer" DAG after after Monday's data has been processed; which > would be on a Tuesday, after a run of "Producer" off the execution date of > Monday. > > The last task in "Producer" is a TriggerDagRunOperator that only triggers > IF it notices that now() is Tuesday. This triggers an instance at the right > time of a child DAG of the "Consumer" type. > > > On Mon, Jun 13, 2016 at 11:50 AM, Rafael Barbosa <[email protected]> > wrote: > > > Hi everyone, > > > > I tried looking around, but I cannot find an archive of this mailing > list, > > so I apologize if this question has been asked before. > > > > I am looking for a why to set the dependency between two DAGs: a > "Producer" > > runs every day, appending data in a DB and a "Consumer" that runs weekly, > > generating a summary report of the new data. > > > > I would like to run the "Consumer" every Monday (for instance), but only > > after the Monday instance of the daily "Producer" is complete. What would > > be the best way to accomplish this? > > > > Thanks, > > Rafael > > >
