lorransr commented on issue #29332:
URL: https://github.com/apache/airflow/issues/29332#issuecomment-2034612299
Hi all,
I'm encountering an issue with the setup and teardown feature in Airflow,
which I believe is related to this issue. When executing a DAG with the
provided setup script, the teardown task does not wait for dependent tasks to
finish before executing. Specifically, if I clear task t1 while task t3 has any
of the following statuses: finished, failed, or upstream_failed, the teardown
task executes immediately without waiting for t1 to complete.
Steps to Reproduce:
Execute the provided DAG setup script.
Clear task t1 while task t3 has one of the mentioned statuses (finished,
failed, upstream_failed).
Observe the behavior of the teardown task.
Expected Behavior:
The teardown task should wait until all dependent tasks, including t1, have
completed before executing.
Actual Behavior:
The teardown task executes immediately after clearing t1, regardless of the
status of task t3.
Setup Script:
```
default_args = {
"owner": "me",
"start_date": "2024-04-03",
"retries": 3,
"retry_delay": timedelta(minutes=1),
}
with DAG(
"GDP_T_est",
default_args=default_args,
max_active_runs=1,
catchup=False,
) as dag:
setup = BashOperator(
task_id="setup",
bash_command="""echo "starting" && sleep 5 && echo "done" """,
)
t1 = BashOperator(
task_id="t1",
bash_command="""echo "t1 starting" && sleep 10 && echo "done" """,
)
t2 = BashOperator(
task_id="t2",
bash_command="""echo "t2 starting" && sleep 5 && echo "done" """,
)
t3 = BashOperator(
task_id="t3",
bash_command="""echo "t3 starting" && sleep 5 && echo "done" """,
)
teardown = BashOperator(
task_id="teardown",
bash_command="""echo "finishing" && sleep 5 && echo "done" """,
)
(setup >> t1 >> t2 >> t3 >> teardown.as_teardown(setups=setup))
```
Additional Information:
Airflow Version: 2.7.1
I'm using the airflow image: apache/airflow:2.7.1
Video:
https://github.com/apache/airflow/assets/41809492/9ad81eee-387b-4b00-b402-5b3065ab5e1f
Is this an expected behavior for this feature at the moment?
It seems problematic, specially when using dbt with cosmos, and we end up
having the teardown of our cluster while some tasks are still running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
