GitHub user potiuk added a comment to the discussion: Occasional Airflow tasks: Upstream tasks succeed and downstream tasks are not scheduled
I guess you are talking about "mini-scheduler" only - your downstream tasks are still getting scheduled but not immediately after task completes. Your downstream tasks will be scheduled by the next scheduler loop I presume. If so. Yes. It might happen. "Mini-scheduler" is an optimisation that is not guaranteed to work. In fact in Airlfow 3 we are removing it altogether because it has proven to be problematic in many ways - one of the problems was that it causes excessive locking on the database - where the database can be locked by - sometimes - many tasks completing at about the same time. So I believe in Airflow 2.8 we implemented a strategy to skip some of that if the contention is too high. Also this has been further improved in 2.9 and 2.10 so you will likely get a bit better behaviour when you upgrade. But still - there is no guarantee it will happen immediately after task completes. GitHub link: https://github.com/apache/airflow/discussions/45540#discussioncomment-11795489 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
