Hey everyone,
Thank you for attending the dev call earlier today. I updated
our meeting notes on the Airflow wiki and the link for those notes is here
<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451975091/2026-09-24+Dev+call+Minutes>
To everyone who attended the meeting, please check the summary and add
anything I may have missed. For those who could not join, please let us
know if you disagree with anything discussed and agreed upon in
the meeting. Also, please ask questions if something is unclear.
Our next meeting is scheduled for Thursday, the 8th of October at the same
time, i.e., 8 a.m. Pacific Time.
If you would like to add anything to the agenda or if I missed anything
from the last call, please let me know
Best regards,
Vikram
--
Below is the summary from the call:
-
Release Updates and Getting Ready for 3.4.0 (Rahul Vats)
-
Rahul outlined the current proposed timeline for 3.4.0 up front: dev
freeze and branch cut on 5th October, followed by Beta 1 and Beta 2, with
final release on 26th October.
-
No 3.3.3 patch is planned given the timeline. Cherry-picking on 3.3
will continue until dev freeze in case a patch is still needed.
Jarek asked
whether cherry-picking could stop now to save overhead. Rahul held off,
wanting the option open until the freeze date.
-
Rahul asked dev owners to write up their own test plans, alongside
automated sanity and end-to-end tests, and to use each AIP's
update on this
call to check progress against the freeze date.
-
Shahar asked whether Common AI 1.0 is a hard dependency of 3.4.
Vikram clarified it's a communication and timing alignment, not a code
dependency, similar to how Go and other SDKs have shipped alongside past
releases. Shahar confirmed a provider wave will land before the 26th and
offered to help with an ad hoc release if needed.
-
Jarek flagged a process change: releases now include a staging sync
step, and parallel releases must avoid triggering it, since it can wipe
in-progress voting documentation from staging.
-
AIP-85: DAG Importer, YAML DAGs, and TypeScript DAGs (Jason Liu, Pierre
Jeambrun, Dilnaz Amanzholova)
-
Jason reported the TypeScript, Go, and Java SDKs all had their user
interfaces refactored about two weeks ago for clarity. Core
integration is
nearly done across all three. Client-side work still needs many
more PRs to
close the feature gap, with TypeScript furthest along. Jason is pairing
with Pierre on TypeScript and with a community contributor, Henry, on Go
and Java, and has gathered feedback from the Go and Java/Spring
communities.
-
Pierre confirmed the core-side changes are nearly done and should
land for 3.4, while client-side work across all languages is
still the bulk
of the remaining effort.
-
On YAML, Dilnaz reported the importer's SDK, interface, and public
API are merged. A follow-on discussion, prompted by TP, Jason,
and others,
proposed also moving DAG-processor forking logic into the
importer itself.
An ADR was written, but the scope proved too large for this release.
Consensus: ship AIP-85 as originally scoped, wiring the importer into the
DAG processor as-is, and continue the ADR discussion post-release. One PR
is open now, with several smaller wiring PRs expected Monday or Tuesday.
-
Ash asked what YAML format Airflow core itself will support, versus
separate formats from Google or DAG Factory. Dilnaz, from the
Google Cloud
Composer team, said Google's own YAML importer will integrate
closely with
Composer and will differ from the OSS one. Separately, Astronomer plans a
YAML importer compatible with DAG Factory. The OSS design doc is expected
soon.
-
Vikram asked for clarity on scope: what DAG syntax YAML will support
at launch (dynamic task mapping, task loops, etc.), and what the
overall UX
looks like for someone running TypeScript DAGs, including setup,
configuration, CLI, and UI, framing this as an adoption risk if
the answers
aren't settled before launch. Jarek agreed some limitations are
necessary,
since fully declarative YAML covering all imperative DAG
patterns tends to
produce YAML no one can write by hand.
-
Consensus: mark the YAML importer experimental for 3.4, gather
feedback, and iterate afterward. Dilnaz agreed this is the right
call given
the ambition of full DAG-syntax coverage in one release. Vikram
asked that
TypeScript's own scope questions be closed out by the next call,
given it's
the first real signal that TypeScript DAGs are production-ready and first
impressions matter for adoption.
-
AIP-111: Task Loops and Task Groups (Ash Berlin-Taylor)
-
Task groups have been voted and accepted. Task loops (AIP-113) is
still in draft. Ash is doing data-modeling work to handle task instance
state edge cases before the rest of the implementation can proceed. Ash
expressed concern that the code freeze of 5th October was too tight.
-
Migration impact: tested on 300 million task instance rows, taking
about 30 minutes on a small R5 instance, treated as an upper bound.
Scheduler impact is minimal since the API server does most of
the work. UI
navigation for loops still needs refinement, per Brent's ongoing work.
-
Ash flagged a related bug found along the way: UUIDs introduced in
3.0 for task instance IDs were being reassigned on retry rather than
staying immutable. A fix is in draft, and an existing, AI-generated
community PR attempting the same fix will be closed as redundant.
User-visible effect: retry state will now correctly show as a second
attempt rather than reusing the first attempt's ID.
-
AIP-104: Task Iteration and Batch Tasks (Amogh Rajesh Desai, for David
Blain)
-
David was not on the call. Amogh has been reviewing and iterating
with him since the last call, including resolving naming
confusion around a
parameter, batch size, that David has already addressed.
-
The PR is roughly 7,000 lines. Ash and Amogh clarified it bundles two
largely separable features: pass-through dynamic iteration, which lets
downstream mapped tasks be scheduled before the producer task completes,
and batch tasks, which can already land independently and don't depend on
the iteration work.
-
Consensus: ask David to split this into stacked PRs so batching can
land on its own. Jarek noted this had already come up in a Slack
discussion.
-
AIP-104 in its entirety is unlikely to land in 3.4.
-
AIP-94: Decouple Remote Commands from Airflow CLI (Bugra Ozturk)
-
Vikram asked whether this is actively being worked on. Rahul said
there's been no recent activity since a discussion two calls ago, and
agreed to connect with Bugra directly and remove it from the milestone if
it's stalled.
-
Common AI Provider 1.0 (Kaxil Naik)
-
Kaxil reported the sandboxing, interface cleanup, and pluggability
work discussed on the last call has largely merged. Batch APIs have an
initial interface merged and tested with Anthropic and OpenAI.
Vikram noted
this was one of the most requested capabilities he's heard from users.
-
Kaxil plans to defer memory support to agent frameworks that already
support it.
-
Docs have been revamped and will be visible once the current RC vote
passes.
-
Kaxil highlighted a recent feature: classification support using Jev,
TypeSafe AI's System One model, for LLM branch operators and retry
policies, which performed well on accuracy and latency in his testing. A
retry chain policy also shipped, letting users chain fallback
behavior, for
example fast-retrying networking errors versus not retrying auth errors,
falling back to Jev classification, then rule-based classification, then
default retry delay. This was also backported to 3.1 and 3.2 via
the compat
module.
-
Vikram said the provider doesn't look ready for a 1.0 RC yet and
reads more like a 0.9. Kaxil confirmed the current release is
0.10 RC1, cut
by Shahar, and that more cleanup is needed before he would cut a
1.0 beta.
Ash noted Common AI isn't tied to the 3.4 freeze, so it has more runway
than the rest of 3.4's scope.
-
AWS Managed Agent and Google ADK integrations are in progress. Kaxil
will reach out to Niko or another AWS contact, and separately to
someone on
the Google team, to align on the agent invocation interface.
-
AIP-118: Self-Hosted CI Runners (Shahar Epstein, flash talk)
-
Shahar is running a large providers release, split into two GitHub
issues due to length, mostly automated, and asked for help testing Common
AI, Google, Amazon, and Fab providers specifically.
-
On CI: Shahar has tested a self-hosted setup on AWS using Terraform
and Kubernetes to offload some CI runs. Since Summit, Shahar identified a
tool called "Runs On" that turns this into a one-click setup,
already used
by the ASF, Apache Arrow, and DataFusion. Shahar has spoken with its CEO
and plans to test it over the next few days, aiming to finalize
the AIP and
reduce load on existing GitHub-hosted runners.
-
Ash noted the project has $100,000 in Azure credits, expiring in
about two years, as a possible alternative or complement to the AWS-based
approach. Shahar said an earlier attempt to use the Azure access hadn't
worked and Runs On currently only supports AWS, but agreed it's worth
revisiting. Jarek suggested starting with Runs On now and layering in the
more complex Kubernetes-based setup later, since the two aren't mutually
exclusive.
-
Discussion Topics:
-
Maintainer Time, PR Limits, CI in Forks, Self-Hosted Runners,
Airflow-Backed Maintainer AI Assistance was not reached due to time
constraints.
Action Items
-
Amogh to ask David to split the AIP-104 PR into stacked PRs so batching
can land independently of dynamic iteration.
-
Rahul to check in with Bugra on AIP-94 and remove it from the milestone
if it's still inactive.
-
Kaxil to reach out to the AWS and Google teams on Managed Agent and ADK
integration alignment.
-
Shahar to test the "Runs On" self-hosted runner setup over the next few
days and progress AIP-118.
-
Pierre and Jason to close out open TypeScript DAG scope and UX questions
before the next call.
-
Community to help test the current large providers release, especially
Common AI, Google, Amazon, and Fab.
Vikram Koka
Chief Strategy Officer
Email: [email protected]
<https://www.astronomer.io/>
