SamWheating opened a new pull request, #29909: URL: https://github.com/apache/airflow/pull/29909
Re: https://github.com/apache/airflow/issues/29900 This introduces a new `@continuous` Timetable which will always try to start new DAGRuns. The degree of parallelism can be then bounded with the `max_active_dagruns` parameter. This is a little bit different from the currently available approaches: - It doesn't have the notion of catchup like `schedule_interval="* * * * *"` and can hypothetically run more often - Its a lot lighter-weight (both in definition and execution) than using a TriggerDagOperator at the end of a DAG. Tested in Breeze with the following DAG: ```python from airflow.models import DAG from airflow.operators.bash import BashOperator from datetime import datetime from airflow.models.param import Param dag = DAG( f"continuous_dag", schedule_interval="@continuous", start_date=datetime(2021, 1, 1), ) task = BashOperator(task_id="the_task", dag=dag, bash_command="sleep 10", owner="nobody!") ``` And it seemed to work well: <img width="1893" alt="image" src="https://user-images.githubusercontent.com/16950874/222838739-9c236abb-0fab-49f0-8fb8-c7de18212616.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/16950874/222838739-9c236abb-0fab-49f0-8fb8-c7de18212616.png"](https://user-images.githubusercontent.com/16950874/222838739-9c236abb-0fab-49f0-8fb8-c7de18212616.png"){kind=link}