Thanks, this re-drafted version looks interesting. I’m trying to internalise and understand the new proposal. I’l leave a few comments/questions on the confluence page as I go.
(I have to say, the move away from epic stored procedure def is a welcome change!) -ash > On 22 Apr 2026, at 14:45, Natanel <[email protected]> wrote: > > Hello community, I want to spark again a discussion that was held in the > past and delayed (due to the focus on releasing airflow 3.2), now that 3.2 > is released, I think it might be a good Idea to bring up the discussion > again. > > Wiki: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits > > In the current situation, tasks may starve in airflow, and in large scale > deployments (hundreds of thousands of tasks (or more) per day) we tend to > experience severe starvation, where a group of tasks may starve other > tasks, not allowing them to run, as described in the wiki. > > After the february devcall, where I have proposed the AIP, a few comments > have arised, and so I had begun to research again about different > scheduling algorithms and I have added to the considerations as part of the > AIP. > > As of now, the state of the AIP is where there are a few ideas proposed > (some of which are pretty similar to each other, while others are quite > different), as the main concern from the devcall was that the approaches > given might not be the best way to solve the issue, as it is a very hard > problem to solve. > > After that, I have made some edits to the AIP and to the propositions, in > order to help decide and clarify the advantages and disadvantages of each > approach. > The current "best approach" can be found here here > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition>, > where the new proposed algorithms are defined here > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs> > . > > In order to continue with the effort, a community consensus needs to be > reached about the preferred solution/solutions, once this is done, it is > possible to go on and implement + stress test the proposed solution/s. > > I would appreciate a review from community members, moreover, I would also > appreciate any new propositions or improvements which can be done. > > Best regards > Natanel. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
