I probably missed the updates here, but when this was brought up at the dev call a while ago, I thought the response was a firm "No". Did I miss the follow-on updates?
The discussion centered on the "why". This is such a "power user" feature, which is fine, but it lacks articulation of the projected benefit in quantifiable terms. It has very high technical complexity in the core of Airflow, which could cause massive ripple effects and therefore requires a cautious approach. I left a comment in the AIP document as well, about needing to understand the "why" better. Vikram On Thu, May 14, 2026 at 1:28 AM Christos Bisias <[email protected]> wrote: > Hello, > > I've got a question but I can't find a way to add a comment under the AIP. > > The 'Weighted Aging and SLA Urgency' solves the starvation issue but do we > have an idea of what happens with the scheduler's performance? > > Christos > > On Thu, May 14, 2026 at 7:47 AM Elad Kalif <[email protected]> wrote: > > > Kind reminder to everyone who still wants to add comments > > If no further comments i think we can move this to a vote. > > > > On Tue, May 5, 2026 at 12:00 AM Jarek Potiuk <[email protected]> wrote: > > > > > Very interesting - also will take a look shortly - and great how it > seems > > > to tap into SLA urgency indeed.. > > > > > > On Mon, May 4, 2026 at 10:08 PM Przemysław Mirowski <[email protected] > > > > > wrote: > > > > > > > +1 for Weighted Aging and SLA Urgency proposition. > > > > ________________________________ > > > > From: Elad Kalif <[email protected]> > > > > Sent: 30 April 2026 12:08 > > > > To: [email protected] <[email protected]> > > > > Subject: Re: [DISCUSS] AIP-100: Eliminate Scheduler Starvation > > > > > > > > I love the idea of dynamic priorities and I think this is a good > > > direction. > > > > > > > > On Thu, Apr 30, 2026 at 12:58 PM Ash Berlin-Taylor <[email protected]> > > > wrote: > > > > > > > > > Thanks, this re-drafted version looks interesting. I’m trying to > > > > > internalise and understand the new proposal. I’l leave a few > > > > > comments/questions on the confluence page as I go. > > > > > > > > > > (I have to say, the move away from epic stored procedure def is a > > > welcome > > > > > change!) > > > > > > > > > > -ash > > > > > > > > > > > On 22 Apr 2026, at 14:45, Natanel <[email protected]> > wrote: > > > > > > > > > > > > Hello community, I want to spark again a discussion that was held > > in > > > > the > > > > > > past and delayed (due to the focus on releasing airflow 3.2), now > > > that > > > > > 3.2 > > > > > > is released, I think it might be a good Idea to bring up the > > > discussion > > > > > > again. > > > > > > > > > > > > Wiki: > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits > > > > > > > > > > > > In the current situation, tasks may starve in airflow, and in > large > > > > scale > > > > > > deployments (hundreds of thousands of tasks (or more) per day) we > > > tend > > > > to > > > > > > experience severe starvation, where a group of tasks may starve > > other > > > > > > tasks, not allowing them to run, as described in the wiki. > > > > > > > > > > > > After the february devcall, where I have proposed the AIP, a few > > > > comments > > > > > > have arised, and so I had begun to research again about different > > > > > > scheduling algorithms and I have added to the considerations as > > part > > > of > > > > > the > > > > > > AIP. > > > > > > > > > > > > As of now, the state of the AIP is where there are a few ideas > > > proposed > > > > > > (some of which are pretty similar to each other, while others are > > > quite > > > > > > different), as the main concern from the devcall was that the > > > > approaches > > > > > > given might not be the best way to solve the issue, as it is a > very > > > > hard > > > > > > problem to solve. > > > > > > > > > > > > After that, I have made some edits to the AIP and to the > > > propositions, > > > > in > > > > > > order to help decide and clarify the advantages and disadvantages > > of > > > > each > > > > > > approach. > > > > > > The current "best approach" can be found here here > > > > > > < > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition > > > > > >, > > > > > > where the new proposed algorithms are defined here > > > > > > < > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs > > > > > > > > > > > > . > > > > > > > > > > > > In order to continue with the effort, a community consensus needs > > to > > > be > > > > > > reached about the preferred solution/solutions, once this is > done, > > it > > > > is > > > > > > possible to go on and implement + stress test the proposed > > > solution/s. > > > > > > > > > > > > I would appreciate a review from community members, moreover, I > > would > > > > > also > > > > > > appreciate any new propositions or improvements which can be > done. > > > > > > > > > > > > Best regards > > > > > > Natanel. > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > >
