[
https://issues.apache.org/jira/browse/AIRFLOW-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847494#comment-16847494
]
Alejandro commented on AIRFLOW-4553:
------------------------------------
[~bolke] It looks like this behavior was introduced by
[https://github.com/apache/airflow/pull/2888]
We are using MySQL and it looks like only the changes applied to the task
instance within the last iteration of the for loop are really applied. That
means that only the last task gets its state changed to SCHEDULED
Changing the commit back to be within the for loop fixes the issue. Any ideas
why the state is not committed in bulk as expected?
> Scheduler only schedules one task at a time with CeleryExecutor
> ---------------------------------------------------------------
>
> Key: AIRFLOW-4553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4553
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.2, 1.10.3
> Reporter: Alejandro
> Priority: Major
>
> I've recently upgraded from Airflow 1.9.0 to Airflow 1.10.3. There has been a
> change in how the scheduler put tasks up for execution that have affected
> some of our DAGs. We use the CeleryExecutor.
> On Airflow 1.9.0 the scheduler would set as `scheduled` as many tasks as it
> could, For example in a DAG with 5 independent tasks and enough concurrency
> set, the scheduler would queue the 5 tasks for execution and all 5 tasks
> could potentially start running.
> On Airflow 1.10.3 the scheduler would set as `scheduled` only one task at a
> time, For example in a DAG with 5 independent tasks and enough concurrency
> set, the scheduler would queue the 1 task for execution and the other 4 would
> not stay with a None state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
