jerell-mendoza commented on issue #32617:
URL: https://github.com/apache/airflow/issues/32617#issuecomment-1662965740
@hussein-awala sorry for such a late response!
```
with DAG(
dag_id=dag_id,
max_active_runs=1,
start_date=datetime(2020, 8, 5),
schedule=schedule,
catchup=False,
params=params,
default_args=get_dag_defaults(log_pagerduty=False),
tags=["orchestration"],
) as dag:
... (there are several trigger dag run operators here, but here's one)
TriggerDagRunOperator(
task_id="trigger_orchestration_source_etls",
trigger_dag_id="orchestration_source_etls",
wait_for_completion=True,
)
...
```
This is code for `orchestration_source_etls` DAG
```
with DAG(
dag_id=dag_id,
schedule=schedule,
max_active_runs=1,
max_active_tasks=max_parallelism,
catchup=False,
params=params,
default_args=get_dag_defaults(log_pagerduty=False),
tags=tags,
) as dag:
orchestrated_weighted_dags = get_weighted_dags(orchestrated_dag_ids)
for etl_dag in orchestrated_weighted_dags:
dag_name = etl_dag["dag_name"]
TriggerDagRunOperator(
task_id=f"trigger_{dag_name}",
trigger_dag_id=dag_name,
priority_weight=etl_dag["weight"],
weight_rule="absolute",
wait_for_completion=True,
)
```
Sorry trying not to reveal too much of our code - but in essence it's
Top Level Dag =>
Triggers Dags [a, b, c, d, etc] =>
[
Dag a => Triggers Dags [e, f, g, etc]
Dag b => Triggers Dags [i, j, k, etc]
]
If I have time maybe I can copy and generalize our code more and see if I
can reproduce it with simple dags that keep triggering other dags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
