GitHub user rraghunathan-exelixis added a comment to the discussion: Airflow
unable to handle more than 16 tasks at once
@atusman24 are you able to find out why it is queued state? I'm facing the
similar issue, im running 40 tasks per dag, but it is only running 23-25 tasks
per run, all others are struck in queued state. There is not proper logs on any
of the components.
these are my values,
celery:
worker_concurrency: 50
core:
parallelism: 128
max_active_tasks_per_dag: 50
max_active_runs_per_dag: 50
Please let me know
GitHub link:
https://github.com/apache/airflow/discussions/32609#discussioncomment-14167408
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
