GitHub user FelipeRamos-neuro added a comment to the discussion: Expand dataset (asset) scheduling to tasks as well, changing the dag run to a deferred state while it awaits for events triggered externally
Today, Airflow supports asynchronous execution through deferrable operators, but it is not entirely async because it still relies on a pull-based mechanism and that requires the dag_run to be on a running state, even if the task is deferred to the triggerer component. The proposal is that we could implement a feature to support a push-based mechanism using assets as dependencies for tasks to be run after a certain event is triggered and "stop" the dag, to a deferred state (if necessary), while it awaits for this external communication. I also just found [AIP-82](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow), which attempts to improve external event driven scheduling. GitHub link: https://github.com/apache/airflow/discussions/44816#discussioncomment-11523573 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
