antontimenko edited a comment on issue #14205: URL: https://github.com/apache/airflow/issues/14205#issuecomment-789903499
@zachliu this is an important thing for me, because in production setup we use Airflow sometimes to run lot's of "jobs" (DAG runs) with different configs. For example we sometimes need to run 7000 dag runs with different parameters in config, so these DAG runs are created using some kind of code that makes requests to Airflow HTTP API and triggers these runs. The tasks inside DAG runs use another external APIs that are unreliable, so the ability to retry tasks with some interval is a great feature for us In 1.10.12 there was no issue, scheduler picks 16 runs with most recent execution dates, runs them, and when they are finished it picks another ones, and so on until there is no more dag runs. If something failed, it will just retry let's say in 10 min and will be finished Airflow 2.0 brings great speed up in this process because of new micro schedulers inside workers, which is awesome and now there is a possibility to greatly speed up this process, which is why we upgraded. But this issue makes everything broken and I need to give a look sometimes to Airflow and perform some manual steps to exit deadlock if it occurs :( ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
