+1 for Weighted Aging and SLA Urgency proposition. ________________________________ From: Elad Kalif <[email protected]> Sent: 30 April 2026 12:08 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] AIP-100: Eliminate Scheduler Starvation
I love the idea of dynamic priorities and I think this is a good direction. On Thu, Apr 30, 2026 at 12:58 PM Ash Berlin-Taylor <[email protected]> wrote: > Thanks, this re-drafted version looks interesting. I’m trying to > internalise and understand the new proposal. I’l leave a few > comments/questions on the confluence page as I go. > > (I have to say, the move away from epic stored procedure def is a welcome > change!) > > -ash > > > On 22 Apr 2026, at 14:45, Natanel <[email protected]> wrote: > > > > Hello community, I want to spark again a discussion that was held in the > > past and delayed (due to the focus on releasing airflow 3.2), now that > 3.2 > > is released, I think it might be a good Idea to bring up the discussion > > again. > > > > Wiki: > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits > > > > In the current situation, tasks may starve in airflow, and in large scale > > deployments (hundreds of thousands of tasks (or more) per day) we tend to > > experience severe starvation, where a group of tasks may starve other > > tasks, not allowing them to run, as described in the wiki. > > > > After the february devcall, where I have proposed the AIP, a few comments > > have arised, and so I had begun to research again about different > > scheduling algorithms and I have added to the considerations as part of > the > > AIP. > > > > As of now, the state of the AIP is where there are a few ideas proposed > > (some of which are pretty similar to each other, while others are quite > > different), as the main concern from the devcall was that the approaches > > given might not be the best way to solve the issue, as it is a very hard > > problem to solve. > > > > After that, I have made some edits to the AIP and to the propositions, in > > order to help decide and clarify the advantages and disadvantages of each > > approach. > > The current "best approach" can be found here here > > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition > >, > > where the new proposed algorithms are defined here > > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs > > > > . > > > > In order to continue with the effort, a community consensus needs to be > > reached about the preferred solution/solutions, once this is done, it is > > possible to go on and implement + stress test the proposed solution/s. > > > > I would appreciate a review from community members, moreover, I would > also > > appreciate any new propositions or improvements which can be done. > > > > Best regards > > Natanel. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
