Hi airflow dev team, We have a subdag which looks like the following...
This subdag has a concurrency limit of 8. As you can see 8 tasks after our 'select_locations' task succeed and so do their downstream tasks. The rest of the tasks seem to get forgotten about by the scheduler. We've re-ran this dag a few times, cleaned out the database, ran it again but we still run into the problem. After some digging, we found that the scheduler seems to be adding too many tasks to our redis queue. The worker then picks up these tasks from the redis queue, runs 8 of them fine and fails the rest. Here is the error message from the failed ones : Dependencies not met for <TaskInstance: performance_kit_180_v1.calculate_stats.transform_tripadvisor_reviews 2017-06-21 00:00:00 [queued]>, dependency 'Task Instance Slots Available' FAILED: The maximum number of running tasks (performance_kit_180_v1.calculate_stats) for this task's DAG '8' has been reached. Then is prints this message : -------------------------------------------------------------------------------- FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt 1 of 2. State set to NONE. -------------------------------------------------------------------------------- Queuing into pool None We noticed in the airflow code that this error message wasn't expected to happen and that it might point to a problem in the scheduler. Here is the comment which led us to believe that : # FIXME: we might have hit concurrency limits, which means we probably # have been running prematurely. This should be handled in the # scheduling mechanism. We have other dags which are limited by the concurrency limit just fine and none of the tasks of those dags print the above error on our worker node. Thanks for any insight into this problem!
