+1 (non-binding) Would be very convenient not having to clear out create_cluster and terminate_cluster tasks when needing to re-run one task that does a spark-submit.
On Sun, Jan 15, 2023 at 3:37 AM Ash Berlin-Taylor <a...@apache.org> wrote: > The key thing that isn't possible is the automatic clearing. > > If I notice a problem in the data after the cluster has been torn down and > I want to the rerun a single task then you have to manually clear the > create cluster task and the task you want to run AND the teardown task. > (This is not a hypothetical but a real issue I have had) > > More generally this is part of the trend of giving DAG authors the ability > to specify "directly" the constructs and flows that they currently have to > hack around. > > > -a > > On 15 January 2023 00:26:31 GMT, Alexander Shorin <kxe...@gmail.com> > wrote: >> >> Hi! >> >> This looks like sugar without any practical difference from explicitly >> defined setup/teardown operators in dag. And the example doesn't add any >> automatic'sation except to set you free from defining explicit relations >> between setup, teardown and regular operators, but why all of this when you >> already could do this explicitly and reliable? Looks like a feature just >> for a feature. >> >> If that example of implementation could hide within a context details >> about ti.xcom_pull(task_id="create_cluster") and hide the need of >> explicit call setup/teardown operators - that would be a bit helpful, but >> there is no such a thing. >> >> -- >> ,,,^..^,,, >> >> >> On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor <a...@apache.org> wrote: >> >>> Hello everyone, >>> >>> I am calling for a vote on AIP-52 >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks >>> There haven't been any notable changes to the original document, mostly >>> just clairifications to the proposal. >>> >>> This is my +1, and the vote will last until 10am (UTC) on Monday 16th >>> January. >>> >>> Thanks, >>> Ash >>> (To remind people what this is about, here is the example from the first >>> dicuss email) >>> >>> ``` >>> from airflow import DAG, task, setup, teardown >>> >>> >>> with DAG(dag_id='test'): >>> @setup >>> def create_cluster(): >>> ... >>> return cluster_id >>> >>> @task >>> def load(ti): >>> # Example: >>> cluster_id = ti.xcom_pull(task_id="create_cluster") >>> >>> def summarize(): >>> ... >>> >>> @teardown(on_failure_fail_dagrun=False) >>> def teardown_cluster(): >>> ... >>> cluster_id = ti.xcom_pull(task_id="create_cluster") >>> >>> create_cluster() >>> load() >> summarize() >>> teardown_cluster() >>> ``` >>> >>>
