omkar-foss commented on issue #40841: URL: https://github.com/apache/airflow/issues/40841#issuecomment-2245709461
Hey @shai-ikko, based on the steps you provided above, I was able to reproduce this issue in my local Airflow. I tried out your sensor DAG and observed that the dep check function [here](https://github.com/apache/airflow/blob/main/airflow/ti_deps/deps/prev_dagrun_dep.py#L159-L163) returns with a passing status for task instances in simultaneously running dagruns and never reaches the `wait_for_downstream` check [here](https://github.com/apache/airflow/blob/main/airflow/ti_deps/deps/prev_dagrun_dep.py#L190-L199), causing a race condition between the task instances of the dagruns. Eventually one dagrun succeeds while the other fails. I'm attempting to fix this issue in [this PR](https://github.com/apache/airflow/pull/40963) by skipping [this check](https://github.com/apache/airflow/blob/main/airflow/ti_deps/deps/prev_dagrun_dep.py#L159-L163) if the last dagrun is still running, so the `wait_for_downstream` check is reached and failing status is returned where required. I've tested this PR alongwith your sensor DAG and 10+ simultaneous file touches, and it works as expected. You can also try it out, hope this helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
