Same here! I’d love to see these in my DAGs 😊 From: Pierre Jeambrun <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, December 20, 2022 at 7:52 AM To: "[email protected]" <[email protected]> Subject: RE: [EXTERNAL][DISCUSS] AIP-52 Automatic setup and tear down tasks
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello Ash, I also love this. The AIP answered the question I had. Best, Pierre Le mar. 20 déc. 2022 à 04:41, Ping Zhang <[email protected]<mailto:[email protected]>> a écrit : Hi Ash, I love this idea, especially about the `teardown`, which is way cleaner than defining a task as the downstream task of all other tasks with trigger_rule as `all_done`. Thanks, Ping On Fri, Dec 16, 2022 at 8:05 AM Ash Berlin-Taylor <[email protected]<mailto:[email protected]>> wrote: Hi everyone, I'd like to start a discussion about a new feature we'd like to add to Airflow we call "setup and tear down tasks" https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks Lets start with a code example, that if you are familiar with unittest.TestCase I hope will be familiar and you can guess what the effect would be: ``` from airflow import DAG, task, setup, teardown with DAG(dag_id='test'): @setup def create_cluster(): ... return cluster_id @task def load(ti): # Example: cluster_id = ti.xcom_pull(task_id="create_cluster") def summarize(): ... @teardown(on_failure_fail_dagrun=False) def teardown_cluster(): ... cluster_id = ti.xcom_pull(task_id="create_cluster") create_cluster() load() >> summarize() teardown_cluster() ``` (This has been an itch of mine that I've had since I first started using Airflow back in 2017!) We go in to quite a bit of detail about the semantics and behaviours of these new task types, but the tl;dr (copied from the AIP doc) Add a new syntax for marking tasks as setup/teardown that: * Teardown tasks will still run even if the upstream tasks have otherwise failed * Teardown tasks failing don't always result in the DagRun being marked as failed (up to DAG author to choose failure mode) * Automatically clear setup/teardown tasks when clearing a dependent task I'm not sure when we'll start a vote on this due to the holiday season, but we'd like to start working on this in January. -ash
