Hey everyone,
Thank you for attending the dev call on the 7th and apologies for the
delayed notes. I updated our meeting notes on the Airflow wiki and the link
for those notes is here
<https://cwiki.apache.org/confluence/display/AIRFLOW/2026-05-07+Dev+Call+Minutes>
To everyone who attended the meeting, please check the summary and add
anything I may have missed. For those who could not join, please let us
know if you disagree with anything discussed and agreed upon in
the meeting. Also, please ask questions if something is unclear.
Our next meeting is scheduled for tomorrow, the 21st of May at the same
time. It is scheduled for 8 a.m. Pacific Time.
If you would like to discuss a particular topic, please let me know if you
want to add anything to the agenda below:
Airflow 3.3 and related Development Updates
- AIP-103 update - Amogh Desai
- AIP-108 Java-SDK and Coordinator update - Jason Liu / TP
- Typescript update - Shivam Rastogi
- AIP-86 Deadline Alerts update - Sean Ghaeli, Dennis Ferruzzi
- AIP-67 Multi-team update - Niko, Vincent Beck
If you would like to add anything to the agenda or if I missed anything
from the last call, please let me know
Best regards,
Vikram
--
Below is the summary from the call:
- *Catch-up on action items from last call:*
- *Airflow 3.3 wiki update (Vikram)*
- Vikram was not yet on the call at this point. Ash noted he could
not speak to the specific wiki actions and moved on.
- Bugra noted in chat that AIP-94 had been added to the wiki after
the last call.
- *Airflow 3.3 Development Updates:*
- *AIP-103 Task State Management (Amogh Rajesh Desai)*
- Amogh shared that development on AIP-103 began last week, with
three or four PRs already merged. Work completed so far
includes the ORM
and database layer, the execution API and configurations, and
the context
accessor — wiring up the Task SDK with the execution API and
providing task
state through the context object.
- Amogh gave a live demo using a simple Python service simulating
a paginated newsletter API with an artificial downtime endpoint. Two
versions of a DAG were demonstrated: a classic stateless
operator and a new
stateful operator using task state for checkpointing.
- In the stateless version, when the service went down mid-run and
the task retried, it restarted from index 0, repeating all previously
completed API calls. In the stateful version, the task stored
its progress
at the end of each batch, and on retry resumed from where it
left off — in
the demo, from index 26 rather than 0.
- David Blain noted that this is particularly valuable because
XCom is cleared on retry, whereas task state persists across
retries. He
highlighted a specific use case for dynamic task mapping where the new
capability would allow tracking which mapped tasks have
succeeded and which
have not, without needing the deferred mechanism workaround
he currently
uses.
- Vikram confirmed this use case is explicitly covered in the AIP.
- Shahar raised the question of how the stateful nature of a DAG
should be surfaced in the UI, suggesting clear indicators and
potentially a
panel showing current state. Amogh confirmed this is planned
but not yet
implemented.
- Ash clarified that AIP-103 defines multiple state scopes. The
current demo covers task instance scope, which is the
foundational element
for the retry use case. Asset scope is a separate scope being
worked on by
Jake in the context of event-driven scheduling. Other scopes
are envisioned
for future capabilities.
- Vikram noted that the AIP is designed as a foundation for a
broader set of use cases over time.
- Shivam asked in chat whether the DAG scope state is stored in
the Airflow database or could be stored externally such as in
S3. This was
noted for follow-up.
- AIP-103 doc:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-103%3A+Task+State+Management
- *AIP-108 Java SDK and Coordinator update (Jason Liu)*
- Jason gave an update on progress since the last dev call,
walking through the key changes rather than repeating the full demo.
- The most significant change is the public interface for
multi-language task support. The team has settled on using
the existing
queue argument as the identifier for routing tasks to the
appropriate language coordinator, rather than exposing a new
language parameter. This avoids any table schema changes and
requires no new execution API endpoints. The corresponding
configuration
has been renamed accordingly.
- A second change involves the introduction of a Task Instance
Data Transfer Object, needed to expose queue information from the task
instance to the Task SDK world without relying on the shared module.
- The third and most significant change is that all coordinator
implementations will now live in a new distribution rather
than under a
provider. The new distribution will be installed via
apache-airflow-coordinator-java and imported via a lightweight
import utility, without the provider manager overhead. Jarek
welcomed this
as a new pattern for Airflow extensions beyond providers,
noting it could
pave the way for similar separations in other areas such as
log handling.
- Jason also previewed the architecture for Go and other
languages, noting that for Go and Rust the coordinator simply needs to
launch a binary executable rather than a JVM. A draft Go SDK
implementation
is in progress and already able to serve DAG source code from the
executable. The interface is designed to be generic enough for new
languages to adapt quickly.
- Shivam Rastogi shared in chat that he is testing a
TypeScript/Node SDK this week and has opened a draft PR: #66289
<https://github.com/apache/airflow/pull/66289>.
- Jason offered to share slides with a Mermaid diagram detailing
the coordinator flow for DAG parsing and task execution, for
those who want
to understand the architecture in depth.
- There was a brief discussion around the term "coordinator" —
Jens noted it could be confusing, though no better alternative was
proposed. Jarek said the name is appropriate given the role
will expand
over time.
- AIP-108 doc:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-108+Java+Task+SDK+and+the+Language+Coordinator+Layer
- *AIP-107 Direct Queueing from Triggerer (Jens Scheffler)*
- This was not on the original agenda but Jens raised it briefly
at the start of the call.
- Jens noted he had written AIP-107, which addresses the latency
problem for tasks returning from the Triggerer to a worker.
He asked for
community feedback on the dev list, as the AIP had received
no responses so
far.
- Ash said he had a question about it and would look at it. Jarek
also said he would add it to his list.
- The discussion touched on the current limitation that the
Triggerer cannot write XCom directly without going back
through the worker,
and that enabling supervisor-style async support in the
Triggerer may be
the right path forward.
- AIP-107 doc:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-107+Direct+Queueing+from+Triggerer+-+or+other+means+to+prevent+Latency+for+Tasks+exiting+deferred+State
- *AIP-86 Deadline Alerts update (Sean Ghaeli)*
- Sean gave a status update on the remaining scope for AIP-86,
with the goal of completing everything for the 3.3 release.
- The theme is closing the loop on all existing work rather than
adding new features. The one item that may not make 3.3.0 is migrating
existing callback functionality (on-success, on-failure
callbacks) to the
new deadline alerts framework. Everything else is on track.
- Sean asked anyone with questions to reach out on Zoom or Slack.
- AIP-86 doc:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488182
- *AIP-67 Multi-team update (Vincent Beck)*
- Niko was not on the call. Vincent gave the update.
- Vincent walked through the current work on asset event filtering
for multi-team deployments. By default, a DAG scheduled on
assets will only
be triggered by asset events sent by the same team. Vincent
is implementing
a new allowed_producer_teams attribute that allows a DAG to
explicitly opt in to receiving asset events from other teams.
- A design question was raised by Ash around what happens when a
DAG from a non-allowed team updates an asset: does the asset
get updated
but the DAG not triggered, or is the update itself rejected? Jarek
clarified that the asset gets updated, but the trigger does
not fire for
teams that have not allowed that producer. This is consistent with the
existing data model which tracks asset updates and DAG
triggers as separate
entities.
- Ash noted this could create a confusing user experience where an
asset event is visible in the UI but a DAG did not run, and
emphasized the
importance of making this clear in the UI. Jarek agreed this should be
surfaced clearly.
- Ash also raised an edge case around asset watchers, where two
conflicting definitions from different teams on the same
asset ID could
cause unexpected behavior. Jarek acknowledged this is an
existing issue
that teams exacerbate, and agreed it needs to be addressed.
- AIP-67 doc:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
- *Discussion Topics:*
- *MCP (AIP-91) and AI Assistant (AIP-101) introduction (Shahar
Epstein)*
- Shahar gave an introduction and live demo of the work he is
proposing, which he had also presented at the Airflow Town
Hall the week
before.
- The demo showed a prototype AI assistant built into the Airflow
UI, which can answer natural language questions about the
user's Airflow
environment by connecting to an Airflow MCP server and retrieving live
state. The example shown was "how many DAGs do I have," which
returned a
live answer.
- Shahar outlined the motivation: currently users must manually
navigate between DAGs, runs, task instances, logs, configs, and
documentation to troubleshoot issues. An AI assistant with
live Airflow
context would reduce time to diagnosis significantly.
- He noted that Astronomer has already launched its own AI
assistant, and that some teams are building their own third-party
integrations. The question for the community is whether an open-source
alternative under the Apache Software Foundation is warranted.
- Shahar proposed a two-stage approach: first finalize and release
a stable MCP implementation (AIP-91), then build the AI
assistant (AIP-101)
on top of it. He proposed forming a task force or SIG to
drive this work,
with discussion in the existing MCP Slack channel and
potential periodic
video calls.
- Ash asked whether the AI assistant needs to be in Airflow Core
or could be a plugin. Vikram raised a broader challenge: does
any of this
need to be in the Airflow repo at all, or is this better suited as an
ecosystem project? He drew a parallel with DAG builders and tools like
Kubernetes or Celery, which Airflow integrates with but does
not own, and
emphasized the maintainability burden of keeping AI tooling
current given
how rapidly the space is moving.
- Jarek framed the question differently, saying the more important
distinction is whether this is released by the Apache
Software Foundation
PMC, as that carries a trust and brand implication. He also
noted the MCP
security landscape is very active with many known
vulnerabilities being
discovered, and cautioned against moving too fast. He
suggested learning
from what others have got right and wrong before committing.
- Jens said from a UX perspective he sees real value for non-power
users, and suggested the AI assistant could be implemented as
a plugin with
a small extension point added to the UI sidebar, keeping the
footprint in
core minimal. He agreed the MCP question needs to be resolved
first. He
also noted that whatever is built should use a pluggable
model rather than
bundling a specific model, so users can bring their own.
- Brent added in chat that beyond MCP, Airflow should consider a
plugin registry to make plugins more discoverable rather than
adding more
to core.
- The team agreed the MCP question is the foundational one. Shahar
committed to developing a security-focused approach for the MCP and
bringing it back to the group for discussion.
- Jens closed with a joke: "The letter S in MCP stands for
Security."
- AIP-91 doc:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=364349979
- AIP-101 doc:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-101+Airflow+AI+Assistant
Vikram Koka
Chief Strategy Officer
Email: [email protected]
<https://www.astronomer.io/>
