Another option that is somewhat of a hack is to use templating to run a
dummy workload.
So say for for a bash_operator you could have something like:
bash_command = """
{% if dayofweek(execution_date) == 'sunday' %}
./dostuff
{% else %}
echo "Skipping.;."
{% endif %}
"""
Max
On Tue, Jun 14, 2016 at 12:47 AM, Rafael Barbosa <[email protected]>
wrote:
> Hi Chris,
>
> It's not clear to me how to use the ExternalTaskSensor. In the case I
> described, as far as I understand, it can be used for the "consumer" to
> wait for every instance of the "producer" to finish. However, I would like
> the the "consumer" to wait 7 executions of the "producer".
>
> Am I missing something?
>
> Best,
> Rafael
>
> Rafael Barbosa
>
> On Tue, Jun 14, 2016 at 5:52 AM, Chris Riccomini <[email protected]>
> wrote:
>
> > An alternative approach would be to use the ExternalTaskSensor to sense
> > when the task(s)/data that you depend upon has finished/arrived.
> >
> >
> >
> https://pythonhosted.org/airflow/code.html#airflow.operators.ExternalTaskSensor
> >
> > On Mon, Jun 13, 2016 at 9:08 AM, Laura Lorenz <[email protected]>
> > wrote:
> >
> > > We do something like that here here in this way, and I'd love feedback
> > from
> > > anyone else if they do it a different way. Basically we use
> > > TriggerDagRunOperators and a task in the "Producer" that cares what day
> > of
> > > the week it is being run, and a "Consumer" DAG with no schedule
> interval
> > > that depends on being triggers when the right conditions of the
> > "Producer"
> > > DAG are met.
> > >
> > > In more detail, we have a daily DAG like your "Producer" that ingests
> > data
> > > every day. We want to run a series of tasks including generating a
> report
> > > in a "Consumer" DAG after after Monday's data has been processed; which
> > > would be on a Tuesday, after a run of "Producer" off the execution date
> > of
> > > Monday.
> > >
> > > The last task in "Producer" is a TriggerDagRunOperator that only
> triggers
> > > IF it notices that now() is Tuesday. This triggers an instance at the
> > right
> > > time of a child DAG of the "Consumer" type.
> > >
> > >
> > > On Mon, Jun 13, 2016 at 11:50 AM, Rafael Barbosa <[email protected]>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I tried looking around, but I cannot find an archive of this mailing
> > > list,
> > > > so I apologize if this question has been asked before.
> > > >
> > > > I am looking for a why to set the dependency between two DAGs: a
> > > "Producer"
> > > > runs every day, appending data in a DB and a "Consumer" that runs
> > weekly,
> > > > generating a summary report of the new data.
> > > >
> > > > I would like to run the "Consumer" every Monday (for instance), but
> > only
> > > > after the Monday instance of the daily "Producer" is complete. What
> > would
> > > > be the best way to accomplish this?
> > > >
> > > > Thanks,
> > > > Rafael
> > > >
> > >
> >
>
