GitHub user yokwejuste added a comment to the discussion: Tasks in a DAG cannot be executed in parallel to find a solution
Please kindly check your configurations on the `airflow.cfg` file to be sure of your parallelism configs. ```cfg parallelism = 32 max_active_tasks_per_dag = 16 dags_are_paused_at_creation = True max_active_runs_per_dag = 16 max_consecutive_failed_dag_runs_per_dag = 0 ``` Mostly the reasons might be: Cause | Description -- | -- Concurrency limit | Not enough slots (max_active_tasks_per_dag, parallelism, pools) Executor type | Sequential executor only allows one task at a time Worker resources | Not enough workers, CPUs, or memory Queue limits | Limited workers in a queue Dependency | Upstream not finished, trigger rules not met Manual actions | Tasks/dag paused, or waiting for external trigger Scheduler lag | Scheduler is overloaded or slow GitHub link: https://github.com/apache/airflow/discussions/53561#discussioncomment-13823184 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
