paramjeet01 commented on issue #38968: URL: https://github.com/apache/airflow/issues/38968#issuecomment-2059521327
After reviewing various GitHub and Stack Overflow discussions, I've made updates to the following configuration and migrated to Airflow version 2.7.2 with apache-airflow-providers-cncf-kubernetes version 8.0.0: ``` [scheduler] task_queued_timeout : 90 max_dagruns_per_loop_to_schedule : 128 max_dagruns_to_create_per_loop : 128 max_tis_per_query : 1024 ``` Additionally, scaling the scheduler to 8 replicas has notably improved performance. The executor slots being exhausted was solved using max_tis_query to max number. Sorry I couldn't find the root cause of the issue but I hope this helps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
