GitHub user potiuk added a comment to the discussion: Deferred operator causing deadlock and Airflow falled down
I don't think so @bugraoz93 :). This one looks pretty legitimate (and sorry @nesp159de - we recently have been flooded with almost good looking issues generated by AI so we are pretty "sensitive" now). But I am not sure we can do anything about it regardless. There are a number of places where mysql can deadlock - where it is completely unnecessary and we know it - the deadlocks are far less often happening on Postgres and our recommendation - for now - would be to switch to postgres. There are few other things you can do: * try to disable "schedule_after_task_execution" - which is an often source for deadlock * upgrade to latest Airflow 2.10.* - you are running pretty old version of Airlfow and you miss more or less 1000 fixes and new features that have been implemented since. Among them - especially in some of the latest versions of Airlfow was limiting a number of deadlocks that you might have. If you look at the changelog you will be able to find them. Generally speaking "upgrade to latest version" and "check if the problems are still there" is absolutely recommended action before anyones time is spent on looking at this issue. * Airlfow 3 (coming in 2-3 months) will have completely revamped architecture where removing database deadlocs and generally database pressure is one of the most important changes that drove the re-architecting. Stay tuned and upgrede/test it when ready Converting it into discussion if more discussion is needed. GitHub link: https://github.com/apache/airflow/discussions/46069#discussioncomment-11958257 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
