On 2026/10/12 22:52:31 Vikram Koka via dev wrote:
> Thanks, David. It's an interesting question and I don't yet have a
> formulated perspective on this.
>
> As part of this, you raised a topic that I had lost track of, specifically:
> AIP-59.
> Is it actively deployed and being used now?
>
> Best regards,
> Vikram
>
> On Mon, Oct 12, 2026 at 3:42 PM Blain David <[email protected]> wrote:
>
> > Hi all,
> >
> > I'd like to restart the discussion on performance testing. This time
> > it's about a layer that AIP-59 doesn't cover: micro-benchmarks for
> > single hot-path methods, run automatically on pull requests.
> >
> > Why now?
> >
> > While addressing review comments on Task Iteration (#62922 [1]), a fix
> > to CommsDecoder.asend changed it from two thread hops per request to
> > one. Every async SDK call a task makes goes through it: Variables,
> > Connections, XComs, the task state store. Measured locally (median of
> > three runs, before -> after):
> >
> > send, one sync round trip (unchanged) 11.5 us -> 11.4 us (none)
> > asend, one async round trip 108.7 us -> 58.0 us (-47%)
> > asend, 50 concurrent coroutines on a loop 18.6 ms -> 11.7 ms (-37%)
> >
> > The reverse change, an extra thread hop added in passing, would have
> > cost the same amount, and nothing in CI would have noticed. Today we
> > only find such regressions when someone measures by hand (or by agent),
> > often after a release: #71721 [2] reports structured task logging in 3.2.0
> > multiplying high-volume task runtime by 2.64.
> >
> > Micro-benchmarks could relate to AIP-59 (Performance tests framework [3]),
> > but would sit one layer below it, not replace it. They target single
> > methods on
> > hot paths and flag a regression in the PR that introduces it, in seconds
> > and
> > on every PR. AIP-59 answers the question micro-benchmarks can't: how a
> > whole release behaves under load.
> >
> > Draft PR #74562 [8] adds:
> >
> > *
> > task-sdk/benchmarks/, next to tests/ and outside the default test
> > *
> > paths, with three benchmarks of the supervisor comms written as
> > plain pytest functions using pytest-codspeed (MIT);
> > *
> > a "benchmarks" dependency group;
> > *
> > a workflow that runs the benchmarks through CodSpeed in simulation
> > *
> > That mode counts CPU instructions rather than measuring time,
> > so results don't depend on how busy the shared runner is, and
> > CodSpeed comments each PR with its change against main.
> >
> > CodSpeedHQ/action is already on the ASF allow-list (v5.2.1, pinned in
> > apache/infrastructure-actions). It authenticates through OIDC, so no
> > secret is needed. Someone with admin rights would still have to
> > enable the CodSpeed app for apache/airflow.
> >
> > In the meantime
> >
> > Following a suggestion in the contributors Slack, draft PR #74579 [9]
> > adds a "Performance-sensitive components" section to the
> > airflow-publish-changes agent skill. It lists the components that run
> > for every task, SDK call, log line or scheduler loop, and asks agents
> > to measure the changed path against the base commit before pushing
> > and to put the numbers in the PR description. It needs no tooling,
> > and can point at the benchmarks/ folders once they exist.
> >
> > Questions I like your opinion on:
> >
> > 1.
> > Do we want micro-benchmarks for hot paths at all? If so, is this
> > *
> > an extension of AIP-59, or does it need its own AIP?
> > 2.
> > Where should they live and how should they be named? I'd propose
> > *
> > <distribution>/benchmarks/, with paths mirroring the source module
> > *
> > they cover:
> >
> > src/airflow/sdk/execution_time/comms.py
> > -> benchmarks/execution_time/test_comms_benchmark.py
> >
> > CI could then run only the benchmarks covering the files a PR
> > changes.
> >
> > 1.
> > Which tool? pytest-codspeed with the hosted comparison, or
> > *
> > something purely local like pytest-benchmark? The local option
> > *
> > compares against a saved baseline with a threshold, but wall-time
> > *
> > thresholds on shared runners are noisy.
> > 2.
> > What should a regression do: report only (a PR comment), or fail
> > *
> > the check above a threshold?
> > 3.
> > Which paths should come first? Beyond the comms, candidates are
> > *
> > XCom serialisation, template rendering, DAG parsing, and the
> > *
> > scheduler's critical-section queries.
> >
> > [1] https://github.com/apache/airflow/pull/62922
> > [2] https://github.com/apache/airflow/issues/71721
> > [3]
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework
> > [4] https://lists.apache.org/thread/0kyp33cp7q5m98s8q7c3pb6nobhprdyt
> > [5] https://github.com/apache/airflow/pull/41961
> > [6] https://github.com/apache/airflow/pull/41315
> > [7] https://lists.apache.org/thread/kg54mjdj1wg8r6jof28jrbjh7d3p7122
> > [8] https://github.com/apache/airflow/pull/74562
> > [9] https://github.com/apache/airflow/pull/74579
> >
> > Thanks,
> > David
> >
> > General (Internal Property)
> >
>
Hello Vikram, actually I also lost track of it, Ash reminded me of it.
I don't think it's actually implemented...
But my proposal is on a different, more micro level, yet with same ambition to
measure performance.
Kind regards,
David
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
