jscheffl commented on PR #43741: URL: https://github.com/apache/airflow/pull/43741#issuecomment-2460463204
> > In multiple context of high number of concurrent task ( more than 1000 and around 50 new task by sec ) we had problems and by disabling the mini scheduler things are running way more smoothly, > > maybe we could deprecate the option in 2.10.4 and set it to false in 2.11.0 , wdyt ? > > Yeah good point, fancy a PR to 2-10-test? I think deprecating the option as warning would be good. I also see it needs a re-work as with AIP-72 the function can not be "distributed on worker" as in the past. Nevertheless when scheduling MANY tasks this was really a performance boost comparing Airflow v1 to Airflow v2 when this was added. Waiting for scheduler loop to schedule the next is really slowing down DAGs... so maybe we need a (future) similar mechanism to schedule next tasks immediately after one has finished to have the same low-latency like in the past... on the backend of course... (like a notification queue where it makes most-sense to schedule next because one task just finished...) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
