Hello maintainers, I need to have several identical (differing only in arguments) top-level DAGs that can also be triggered together with following constraints / assumptions
- Individual top-level DAGs will have schedule_interval=None as they will only need occasional manual triggering - The series (wiring-up) of DAGs, however, needs to run daily - Order and number of DAGs in series is fixed (known ahead of writing code) and changes rarely (probably once in a month or so) ------- What would be a good strategy to wire-up these top-level DAGs so that they run can be run together as well as independently? (or is there something fundamentally wrong with this idea?) ------- I can think of SubDagOperator (problematic) and TriggerDagRunOperator (satisfactory) but neither appear very tidy. Is there a better way? I am open to complete re-structuring of my workflow as I am yet to deploy it. ------ Here's the link <https://stackoverflow.com/questions/51325525/wiring-top-level-dags-together> to my question on SO. And here are some related links that I went through - Airflow, mark a task success or skip it before dag run <https://stackoverflow.com/questions/50934266/airflow-mark-a-task-success-or-skip-it-before-dag-run> - AIrflow - Splitting DAG definition across multiple files <https://stackoverflow.com/questions/41493605/airflow-splitting-dag-definition-across-multiple-files> - Airflow DAG success callback <https://stackoverflow.com/questions/48164160/airflow-dag-success-callback> - How to set dependencies between DAGs in Airflow? <https://stackoverflow.com/questions/38022323/how-to-set-dependencies-between-dags-in-airflow> - Airflow: When Your DAG is Far Behind The Schedule <https://medium.com/@hafizbadrie/airflow-when-your-dag-is-far-behind-the-schedule-ea11bf02e44c> *Shubham Gupta* Software Engineer zomato
