ephraimbuddy commented on PR #34138: URL: https://github.com/apache/airflow/pull/34138#issuecomment-1709660867
> I think waiting for all upstreams to finish first is the correct behaviour. The documentation also hinted on this. [In the first example](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html#simple-mapping), `sum_it` wouldn’t make sense at all if it does not wait for all upstreams to finish. So this should likely be considered a bug. It also affects normal tasks which makes me see it as a feature. Consider this dag: ```python import time from airflow import DAG from airflow.decorators import task from datetime import datetime from airflow.exceptions import AirflowSkipException with DAG( dag_id="bug", schedule=None, start_date=datetime(2023, 1, 1), ) as dag: @task def task1(): time.sleep(10) @task def task2(): time.sleep(10) @task def task3(): raise AirflowSkipException("skip") @task def task4(): return "task4" [task1(), task2(), task3()] >> task4() ``` When you run the dag, `task3` will skip and that would make `task4` skip. <img width="677" alt="Screenshot 2023-09-07 at 08 55 57" src="https://github.com/apache/airflow/assets/4122866/c2405cf3-8234-4d1a-b429-8f3daa624c92";> @dstandish brought this to my attention and I think we should make it configurable or discuss it in the mailing list if we want to change the behaviour. I would prefer having a configurable behaviour than concluding this in a list and probably changing the behaviour cause someone might depend on this in the wild -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
