Asquator commented on PR #54103: URL: https://github.com/apache/airflow/pull/54103#issuecomment-3769492158
Overall I see it as a nice improvement, because eventually it DOES give us better throughput at the moment. In the long term, we'll probably have to look at a more universal solution that prevents starvation with all the concurrency limits. I would like to see the benchmarks I published in the procedures PR run on this approach. The one thing that bothers me - what if we include both `max_active_tasks` and pool/per task/across dagruns constraints? I'm looking forward to see this merged and continue investigating into more enhancements for the scheduler. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
