+1 non-binding On Sun, Jan 15, 2023 at 9:28 PM Robert Karish <[email protected]> wrote:
> +1 (non-binding) > > Would be very convenient not having to clear out create_cluster and > terminate_cluster tasks when needing to re-run one task that does a > spark-submit. > > On Sun, Jan 15, 2023 at 3:37 AM Ash Berlin-Taylor <[email protected]> wrote: > >> The key thing that isn't possible is the automatic clearing. >> >> If I notice a problem in the data after the cluster has been torn down >> and I want to the rerun a single task then you have to manually clear the >> create cluster task and the task you want to run AND the teardown task. >> (This is not a hypothetical but a real issue I have had) >> >> More generally this is part of the trend of giving DAG authors the >> ability to specify "directly" the constructs and flows that they currently >> have to hack around. >> >> >> -a >> >> On 15 January 2023 00:26:31 GMT, Alexander Shorin <[email protected]> >> wrote: >>> >>> Hi! >>> >>> This looks like sugar without any practical difference from explicitly >>> defined setup/teardown operators in dag. And the example doesn't add any >>> automatic'sation except to set you free from defining explicit relations >>> between setup, teardown and regular operators, but why all of this when you >>> already could do this explicitly and reliable? Looks like a feature just >>> for a feature. >>> >>> If that example of implementation could hide within a context details >>> about ti.xcom_pull(task_id="create_cluster") and hide the need of >>> explicit call setup/teardown operators - that would be a bit helpful, but >>> there is no such a thing. >>> >>> -- >>> ,,,^..^,,, >>> >>> >>> On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor <[email protected]> wrote: >>> >>>> Hello everyone, >>>> >>>> I am calling for a vote on AIP-52 >>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks >>>> There haven't been any notable changes to the original document, mostly >>>> just clairifications to the proposal. >>>> >>>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th >>>> January. >>>> >>>> Thanks, >>>> Ash >>>> (To remind people what this is about, here is the example from the >>>> first dicuss email) >>>> >>>> ``` >>>> from airflow import DAG, task, setup, teardown >>>> >>>> >>>> with DAG(dag_id='test'): >>>> @setup >>>> def create_cluster(): >>>> ... >>>> return cluster_id >>>> >>>> @task >>>> def load(ti): >>>> # Example: >>>> cluster_id = ti.xcom_pull(task_id="create_cluster") >>>> >>>> def summarize(): >>>> ... >>>> >>>> @teardown(on_failure_fail_dagrun=False) >>>> def teardown_cluster(): >>>> ... >>>> cluster_id = ti.xcom_pull(task_id="create_cluster") >>>> >>>> create_cluster() >>>> load() >> summarize() >>>> teardown_cluster() >>>> ``` >>>> >>>>
