> But I think any of such improvements should go
through rigorous performance testing on various shapes of Dags
I've got a test setup that I've been using for a while for performance
testing. It's similar to the integration tests for OTel where the tests
start 1+ schedulers and 1+ workers under breeze as distinct services. I
haven't tried to upstream it because I've found that the performance tests
make sense only when you optimize the scheduler's limits according to the
resources of the machine. There isn't one size fits all. The tests have
been some sort of a template that I keep modifying according to the issue.
I've split the dag folder into 3 subdirectories based on the scenario
1. test heavy load, extremely large linear Dags e.g. 10.000 tasks
2. test different topologies, dags with branching task dependencies, a
root task and then a bunch in parallel, etc.
3. test heavy parallelism, dags with mapped tasks that can expand up to
10.000 tasks running in parallel across all dags
I've also created some sort of resource profiles where each profile sets
the correct configs for the current machine. E.g. The values are different
between my M2 Pro Macbook and my Ubuntu desktop that has twice the CPU and
RAM.
This is an example of profiles
RESOURCE_PROFILES = {
"prof1": {
"worker_concurrency": "5",
"max_tis_per_query": "100",
"parallelism": "100",
"max_active_tasks_per_dag": "8",
"max_active_runs_per_dag": "10",
"default_pool_task_slot_count": "64",
},
"prof2": {
"worker_concurrency": "15",
"max_tis_per_query": "256",
"parallelism": "256",
"max_active_tasks_per_dag": "64",
"max_active_runs_per_dag": "10",
"default_pool_task_slot_count": "256",
},
"prof3": {
"worker_concurrency": str(max(2, get_usable_cpu_count() * 2 // 3)),
"max_tis_per_query": "16",
"parallelism": "4096",
"max_active_tasks_per_dag": "4096",
"max_active_runs_per_dag": "1",
"default_pool_task_slot_count": "4096",
},
Setting the optimal resources and running the tests is only half of it. The
other half is getting the interesting metrics and presenting them in a way
that makes sense for what you are testing. Each time I create new grafana
dashboards according to what I'm trying to validate and understand.
It's not completely hands-off and as simple as start breeze, run it and
then check grafana. But I can try to make it more generic and show a
general picture of Airflow's state and then add some documentation for how
people can extend the profiles and the grafana dashboards to debug a
specific issue.
Please let me know if that will be helpful to exist upstream and I can work
on it.
On Sun, Sep 20, 2026 at 4:54 PM Jarek Potiuk <[email protected]> wrote:
> Yeah. Direction is good. But I think any of such improvements should go
> through rigorous performance testing on various shapes of Dags (I wish we
> had the performance framework in place already). Maybe a good idea is to
> make a prototype of such performance framework carved out from earlier work
> for AIP-59 -
>
> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/278465469/AIP-59+Performance+tests+framework
> and maybe that would be good idea to advance both - performance improvement
> and performance testing ?
>
> On Sun, Sep 20, 2026 at 1:39 PM Przemysław Mirowski <[email protected]>
> wrote:
>
> > +1. It looks good to me, but I don't follow in detail what happens around
> > the scheduler area, so it would be great to get feedback on it from
> > different maintainers.
> >
> > We have had discussions regarding the scheduler performance area for
> quite
> > some time now (e.g., the starvation topic), and the last discussion
> stopped
> > at the need for more debug information from the scheduler (the traces PR
> > change would provide that in some part, at least). Also, in the future,
> it
> > would probably be good to see similar proposals for different Airflow
> core
> > component areas.
> >
> > Thanks for all the work!
> >
> > On 2026/09/10 09:46:47 Christos Bisias wrote:
> > > Hello everyone,
> > >
> > > Just a reminder about this discussion.
> > >
> > > Regards,
> > > Christos
> > >
> > > On Tue, Aug 18, 2026 at 11:26 AM Christos Bisias <
> [email protected]>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > In our production environment, schedulers are often very slow under
> > load
> > > > and my infrastructure team reports extended db locking times.
> > > >
> > > > After investigation and testing, I've come up with an optimization
> > which
> > > > has proved to improve scheduler performance greatly.
> > > >
> > > > Investigation
> > > >
> > > > We already have traces and spans for dag runs to help users
> understand
> > > > what's going on under their tasks and possibly optimize them. So I
> > thought
> > > > why not do the same for Airflow internal operations such as the
> > scheduler
> > > > loop.
> > > >
> > > > I've got an open PR that adds a span for every major step of the
> > > > scheduler's loop iteration.
> > > >
> > > > PR: Add optional debug spans for the scheduler loop
> > > > <https://github.com/apache/airflow/pull/69809>
> > > >
> > > > By using the spans in the above PR, I was able to pin-point the
> > > > performance bottleneck in the part of examining the task instances
> for
> > > > scheduling for a particular dag run.
> > > >
> > > > The link below points to the exact part of the code where the issue
> > lies.
> > > > It's where it fetches all the tasks from the DB and then hydrates
> them
> > into
> > > > ORM objects.
> > > >
> > > >
> > > >
> >
> https://github.com/apache/airflow/pull/69809/changes#diff-aa0338f81a481ca6bc69703521c531d593558d4347a958dc00e5f9689d6802a1R1002-R1007
> > > >
> > > > According to the spans, 10% of that time is spent on the query, which
> > is
> > > > very fast and 90% is spent in the ORM object hydration.
> > > >
> > > > Each iteration of the scheduler loop is linear and all operations are
> > > > taking place in a sequence. We can't get to operation 2 unless
> > operation 1
> > > > finishes. And so, the scheduler scans for tasks that can be queued
> > > > (operation 1) and only after the scan has finished, it sends the
> tasks
> > to
> > > > the workers (operation 2). The more time it takes to scan the tasks,
> > the
> > > > bigger the interval at which tasks are set to QUEUED and picked by
> the
> > > > workers. To explain it in another way, if operation 1 takes 5
> minutes,
> > then
> > > > we will run operation 2 every 5 minutes but if it takes 10 minutes,
> > then we
> > > > will run operation 2 every 10 minutes.
> > > >
> > > > To give you an idea of how heavy the scan is, I ran a test with
> > multiple
> > > > dags, reaching up to 22.000 tasks. For the time needed to execute all
> > these
> > > > tasks, 44% of the entire scheduler work across all iterations was
> spent
> > > > just on hydrating ORM task objects.
> > > >
> > > > Sometimes tasks are running for a while and the scan doesn't do any
> > work,
> > > > but it's still a very important operation that allows the scheduler
> to
> > pick
> > > > up dag run changes quickly. We shouldn't skip it, but we can optimize
> > it.
> > > >
> > > > Proposed approach
> > > >
> > > > The scan fetches all tasks for a dag run from the DB, hydrates ORM
> > objects
> > > > and then splits them into 2 lists of finished and unfinished tasks.
> For
> > > > finished tasks, we don't need full TaskInstance objects because we
> > never
> > > > modify them and we only ever read 5 fields from them. Essentially we
> > are
> > > > wasting computing creating heavy objects we don't need.
> > > >
> > > > If instead of generating the full TaskInstance object for every
> > finished
> > > > task, we just create an immutable lightweight object with only the
> > needed
> > > > fields, performance increases greatly.
> > > >
> > > > Instead of 1 query and then a split, we make the split upfront by
> > having 2
> > > > queries, one for finished tasks which will hydrate the lightweight
> > objects
> > > > and one for unfinished tasks which will hydrate the full TaskInstance
> > > > objects. Everything else stays the same.
> > > >
> > > > Based on gathered metrics, I can see that with the current code in
> > main,
> > > > the workers always have available slots to run tasks and are mostly
> > waiting
> > > > on the scheduler. With the improvement, the scheduler sets tasks to
> > QUEUED
> > > > way faster than the workers can handle. The tasks sit in the queue
> > waiting
> > > > for minutes and the workers are actually becoming the bottleneck.
> > > >
> > > > Here is an open PR with the changes. The PR description contains more
> > info
> > > > regarding testing and screenshots from gathered metrics which display
> > the
> > > > improvement clearly.
> > > >
> > > > PR: Make finished TIs more lightweight when scanning task instances
> for
> > > > scheduling <https://github.com/apache/airflow/pull/71737>
> > > >
> > > > Any feedback is highly appreciated! This proved to be an improvement
> in
> > > > every scenario that I tested. When there isn't much load in the
> > system, the
> > > > difference isn't noticeable. If people could also give it a try to
> make
> > > > sure that it doesn't introduce a regression, that would be great!
> > > >
> > > > Thanks,
> > > > Christos
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
>
