Kenny1217 commented on issue #36920: URL: https://github.com/apache/airflow/issues/36920#issuecomment-2045431878
I'm having the same type of issue running Airflow 2.8.3 on Kubernetes. We're use the Kubernetes executer and use an external Postgres database. We get the same error saying DAG record was locked. The weird thing for us is the dags are running but only allows for one task at a time. So for example say there is 3 dags scheduled at the same time, dag 1 task 1 would execute then dag 2 task 1 then dag 3 task 1 and so on till all the task are completed. So it creates a bottleneck for dags to run. But the thing that locks everything up is if the dag calls the TriggerDagOperator with wait_for_completion set to true. The task would run but get stuck because the dag it triggers will never start because task from the dag that launched is still running. Which completely stops everything from running. Scaling up and down the scheduler fixes it for the most part. It only happens to us like once every two weeks or so its hard to replicate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
