I love the idea. You've clearly put a lot of thought into this so maybe you've already considered and discarded these ideas, but I have two suggestions, for what they are worth. One improvement would be to automagically append/prepend them to the task list. If a task is tagged @setup, then it should be prepended as the first task in the task list. The other suggestion would be that any task decorated with @setup or @teardown (or whatever you decide to name them) would get added to a new setup or teardown taskgroup. The benefit there is that when you get tot he future work of multiple steps, it would make the GUI DAG view cleaner by collapsing all of the setup and teardown tasks by default if there are multiple tasks in either category.
________________________________ From: Ash Berlin-Taylor <[email protected]> Sent: Friday, December 16, 2022 8:04 AM To: [email protected] Subject: [EXTERNAL] [DISCUSS] AIP-52 Automatic setup and tear down tasks CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi everyone, I'd like to start a discussion about a new feature we'd like to add to Airflow we call "setup and tear down tasks" https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks Lets start with a code example, that if you are familiar with unittest.TestCase I hope will be familiar and you can guess what the effect would be: ``` from airflow import DAG, task, setup, teardown with DAG(dag_id='test'): @setup def create_cluster(): ... return cluster_id @task def load(ti): # Example: cluster_id = ti.xcom_pull(task_id="create_cluster") def summarize(): ... @teardown(on_failure_fail_dagrun=False) def teardown_cluster(): ... cluster_id = ti.xcom_pull(task_id="create_cluster") create_cluster() load() >> summarize() teardown_cluster() ``` (This has been an itch of mine that I've had since I first started using Airflow back in 2017!) We go in to quite a bit of detail about the semantics and behaviours of these new task types, but the tl;dr (copied from the AIP doc) Add a new syntax for marking tasks as setup/teardown that: * Teardown tasks will still run even if the upstream tasks have otherwise failed * Teardown tasks failing don't always result in the DagRun being marked as failed (up to DAG author to choose failure mode) * Automatically clear setup/teardown tasks when clearing a dependent task I'm not sure when we'll start a vote on this due to the holiday season, but we'd like to start working on this in January. -ash
