GitHub user leninkumar-sv-office closed a discussion: Max tasks across dags stuck at 487
I have the airflow running on AKS cluster with just the kubernetes executor. The following configs are set for attaining the max concurrency. 1. Parallelism: 128 (per scheduler) 2. No of Schedulers: 10 3. Default pool size: 1200 4. DB: Postgres 5. Airflow Version: 2.8.1 6. Platform: Ubuntu We are running big data workloads and the number of Max task runs can reach 1000. However, with the above configurations, the Max task runs across all the dags is stuck at the magic number 487 and the remaining ones are queued. I have checked the docs about any other parameters that need to be set, however only these are mentioned. Appreciate the suggestions from the community to identify the missing configuration. Thanks GitHub link: https://github.com/apache/airflow/discussions/44244 ---- This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
