Re: Tasks scheduled but never run

2020-03-25 Thread kumar pavan
Hi , We have faced the similar issue and debugged the scheduler code and found that back fill don't have much options given( Back fill logic is different in more scheduling point) .In this case once the DAG is stuck it is stuck . Instead of using backfill you can use trigger_dag which has more l

Tasks scheduled but never run

2020-03-25 Thread heng gu
I was doing backfill running a one task dag. There were over 1000 dag_runs ready to run, I set concurrency=16 and max_active_run=16, I use Celery executor and have 12 workers. Task instances were quickly scheduled, but they stuck there and never get run. I did a search online and got this: [jira