Hello Ash, I also love this. The AIP answered the question I had.
Best, Pierre Le mar. 20 déc. 2022 à 04:41, Ping Zhang <[email protected]> a écrit : > Hi Ash, > > I love this idea, especially about the `teardown`, which is way cleaner > than defining a task as the downstream task of all other tasks with > trigger_rule as `all_done`. > > > Thanks, > > Ping > > > On Fri, Dec 16, 2022 at 8:05 AM Ash Berlin-Taylor <[email protected]> wrote: > >> Hi everyone, >> >> I'd like to start a discussion about a new feature we'd like to add to >> Airflow we call "setup and tear down tasks" >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks >> >> Lets start with a code example, that if you are familiar with >> unittest.TestCase I hope will be familiar and you can guess what the >> effect would be: >> >> ``` >> from airflow import DAG, task, setup, teardown >> >> >> with DAG(dag_id='test'): >> @setup >> def create_cluster(): >> ... >> return cluster_id >> >> @task >> def load(ti): >> # Example: >> cluster_id = ti.xcom_pull(task_id="create_cluster") >> >> def summarize(): >> ... >> >> @teardown(on_failure_fail_dagrun=False) >> def teardown_cluster(): >> ... >> cluster_id = ti.xcom_pull(task_id="create_cluster") >> >> create_cluster() >> load() >> summarize() >> teardown_cluster() >> ``` >> >> (This has been an itch of mine that I've had since I first started using >> Airflow back in 2017!) >> >> We go in to quite a bit of detail about the semantics and behaviours of >> these new task types, but the tl;dr (copied from the AIP doc) >> >> Add a new syntax for marking tasks as setup/teardown that: >> >> * Teardown tasks will still run even if the upstream tasks have >> otherwise failed >> * Teardown tasks failing don't always result in the DagRun being marked >> as failed (up to DAG author to choose failure mode) >> * Automatically clear setup/teardown tasks when clearing a dependent >> task >> >> I'm not sure when we'll start a vote on this due to the holiday season, >> but we'd like to start working on this in January. >> >> -ash >> >
