halilduygulu edited a comment on issue #19622: URL: https://github.com/apache/airflow/issues/19622#issuecomment-975789786
Hi I am having the same case I think. airflow version 2.2.2 , just upgraded from 2.0.2 A huge dag running with concurrency = 10, has 435 tasks and there are other dags which their last tasks are in scheduled state forever now. I increased max_tis_per_query = 300 , it was 100, but did not see any improvement. Weird thing is if I ran another dag it runs fine until last task, then stuck, now I got 5 dags waiting in last step with plenty workers idle. scheduler logs are keep looping over 20 task instace names from big dag. my config ``` parallelism = 32 max_active_tasks_per_dag = 14 max_active_runs_per_dag = 10 max_tis_per_query = 300 ``` what can I do for a workaround now? edit: when I paused this huge dag, immediately other dags' task instances were running and then completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
