Yeah. Direction is good. But I think any of such improvements should go through rigorous performance testing on various shapes of Dags (I wish we had the performance framework in place already). Maybe a good idea is to make a prototype of such performance framework carved out from earlier work for AIP-59 - https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/278465469/AIP-59+Performance+tests+framework and maybe that would be good idea to advance both - performance improvement and performance testing ?
On Sun, Sep 20, 2026 at 1:39 PM Przemysław Mirowski <[email protected]> wrote: > +1. It looks good to me, but I don't follow in detail what happens around > the scheduler area, so it would be great to get feedback on it from > different maintainers. > > We have had discussions regarding the scheduler performance area for quite > some time now (e.g., the starvation topic), and the last discussion stopped > at the need for more debug information from the scheduler (the traces PR > change would provide that in some part, at least). Also, in the future, it > would probably be good to see similar proposals for different Airflow core > component areas. > > Thanks for all the work! > > On 2026/09/10 09:46:47 Christos Bisias wrote: > > Hello everyone, > > > > Just a reminder about this discussion. > > > > Regards, > > Christos > > > > On Tue, Aug 18, 2026 at 11:26 AM Christos Bisias <[email protected]> > > wrote: > > > > > Hello, > > > > > > In our production environment, schedulers are often very slow under > load > > > and my infrastructure team reports extended db locking times. > > > > > > After investigation and testing, I've come up with an optimization > which > > > has proved to improve scheduler performance greatly. > > > > > > Investigation > > > > > > We already have traces and spans for dag runs to help users understand > > > what's going on under their tasks and possibly optimize them. So I > thought > > > why not do the same for Airflow internal operations such as the > scheduler > > > loop. > > > > > > I've got an open PR that adds a span for every major step of the > > > scheduler's loop iteration. > > > > > > PR: Add optional debug spans for the scheduler loop > > > <https://github.com/apache/airflow/pull/69809> > > > > > > By using the spans in the above PR, I was able to pin-point the > > > performance bottleneck in the part of examining the task instances for > > > scheduling for a particular dag run. > > > > > > The link below points to the exact part of the code where the issue > lies. > > > It's where it fetches all the tasks from the DB and then hydrates them > into > > > ORM objects. > > > > > > > > > > https://github.com/apache/airflow/pull/69809/changes#diff-aa0338f81a481ca6bc69703521c531d593558d4347a958dc00e5f9689d6802a1R1002-R1007 > > > > > > According to the spans, 10% of that time is spent on the query, which > is > > > very fast and 90% is spent in the ORM object hydration. > > > > > > Each iteration of the scheduler loop is linear and all operations are > > > taking place in a sequence. We can't get to operation 2 unless > operation 1 > > > finishes. And so, the scheduler scans for tasks that can be queued > > > (operation 1) and only after the scan has finished, it sends the tasks > to > > > the workers (operation 2). The more time it takes to scan the tasks, > the > > > bigger the interval at which tasks are set to QUEUED and picked by the > > > workers. To explain it in another way, if operation 1 takes 5 minutes, > then > > > we will run operation 2 every 5 minutes but if it takes 10 minutes, > then we > > > will run operation 2 every 10 minutes. > > > > > > To give you an idea of how heavy the scan is, I ran a test with > multiple > > > dags, reaching up to 22.000 tasks. For the time needed to execute all > these > > > tasks, 44% of the entire scheduler work across all iterations was spent > > > just on hydrating ORM task objects. > > > > > > Sometimes tasks are running for a while and the scan doesn't do any > work, > > > but it's still a very important operation that allows the scheduler to > pick > > > up dag run changes quickly. We shouldn't skip it, but we can optimize > it. > > > > > > Proposed approach > > > > > > The scan fetches all tasks for a dag run from the DB, hydrates ORM > objects > > > and then splits them into 2 lists of finished and unfinished tasks. For > > > finished tasks, we don't need full TaskInstance objects because we > never > > > modify them and we only ever read 5 fields from them. Essentially we > are > > > wasting computing creating heavy objects we don't need. > > > > > > If instead of generating the full TaskInstance object for every > finished > > > task, we just create an immutable lightweight object with only the > needed > > > fields, performance increases greatly. > > > > > > Instead of 1 query and then a split, we make the split upfront by > having 2 > > > queries, one for finished tasks which will hydrate the lightweight > objects > > > and one for unfinished tasks which will hydrate the full TaskInstance > > > objects. Everything else stays the same. > > > > > > Based on gathered metrics, I can see that with the current code in > main, > > > the workers always have available slots to run tasks and are mostly > waiting > > > on the scheduler. With the improvement, the scheduler sets tasks to > QUEUED > > > way faster than the workers can handle. The tasks sit in the queue > waiting > > > for minutes and the workers are actually becoming the bottleneck. > > > > > > Here is an open PR with the changes. The PR description contains more > info > > > regarding testing and screenshots from gathered metrics which display > the > > > improvement clearly. > > > > > > PR: Make finished TIs more lightweight when scanning task instances for > > > scheduling <https://github.com/apache/airflow/pull/71737> > > > > > > Any feedback is highly appreciated! This proved to be an improvement in > > > every scenario that I tested. When there isn't much load in the > system, the > > > difference isn't noticeable. If people could also give it a try to make > > > sure that it doesn't introduce a regression, that would be great! > > > > > > Thanks, > > > Christos > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
