[
https://issues.apache.org/jira/browse/AIRFLOW-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15565570#comment-15565570
]
Laura Lorenz commented on AIRFLOW-564:
--------------------------------------
For us this hasn't leaked into the problem as described here, but we do have an
execution_date aware DAG that is triggered via another DAG's
TriggerDagRunOperator, and we have to send the parent DAG's execution_date as
part of the payload to its child. I can see how it could escalate into the
problem [~martingrayson] describes if we threw an ExternalTaskSensor into the
mix. I think it makes a lot more intuitive sense for the child DAG's DagRun to
inherit the execution_date of its parent DAG, not one that is mirrored by the
DagRun's start_date.
> DagRunOperator and ExternalTaskSensor Incompatibility
> -----------------------------------------------------
>
> Key: AIRFLOW-564
> URL: https://issues.apache.org/jira/browse/AIRFLOW-564
> Project: Apache Airflow
> Issue Type: Task
> Components: DagRun
> Affects Versions: Airflow 1.7.1.3
> Reporter: Martin Grayson
> Attachments: 1.controller.png, 2. par.png, 3. dep.png, dep_dag.py,
> parent_dag.py, test_dag_scheduler.py
>
>
> I have an hourly Dag that acts to orchestrate other dags, triggering dag runs
> if a precondition is met.
> The controller dag (Dag CD) uses a `TriggerDagRunOperator` operator to do
> this, launching 2 other dags (Dag A, Dag B) in my example (10+ in my real
> life).
> Within these dags they have their own dependencies, for example Dag B is
> dependent on Dag A's completion. This is handled using a
> `ExternalTaskSensor`.
> Since Dag A and B are executed dynamically with the `TriggerDagRunOperator`,
> they're assigned very granular execution dates, rather than dd-mm-yyyy
> hh:00:00 as the scheduler would assign. Due to this, the
> `ExternalTaskSensor` will not succeed unless Dag A and B managed to have
> exactly the same execution_date (down to the second).
> As far as I can tell, I'm not able to force the `ExternalTaskSensor` to poke
> for the correct execution date. The only possible solution would be to
> implement a `execution_date_fn` that looked through the metadata db to find
> the exact execution time. This seems massively complicated for a relatively
> simple requirement.
> Can this scenario be modelled with these components?
> I've attached a set of example dags and some screenshots, showing the issue.
> Thanks,
> Martin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
