Hey everyone,
Thank you for attending the dev call on the 16th. I updated
our meeting notes on the Airflow wiki and the link for those notes is here
<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/444334604/2026-07-30+Dev+call+notes>
To everyone who attended the meeting, please check the summary and add
anything I may have missed. For those who could not join, please let us
know if you disagree with anything discussed and agreed upon in
the meeting. Also, please ask questions if something is unclear.
Our next meeting is scheduled for Thursday, the 13th of August at the same
time i.e. for 8 a.m. Pacific Time.
If you would like to add anything to the agenda or if I missed anything
from the last call, please let me know
Best regards,
Vikram
--
Below is the summary from the call:
-
Catchup on Action Items
-
Release Updates: 3.3.1 (Rahul Vats)
-
Rahul noted RC1 is targeted for 3rd or 4th August, with release
around 10th August. This leaves a 5-6 day testing window that
accommodates
an RC2 if one is needed.
-
Rahul noted this gives roughly 15-20 days of buffer before Airflow
Summit.
-
Rahul reported the current milestone shows 48 open items, but
clarified this is inflated: 40-50% of the count is backport PRs, some
already merged into main.
-
Rahul is reviewing PRs he's comfortable with directly and
requesting code owner reviews for the rest, and to stay
cautious per the
existing release process.
-
Jarek noted he will kick off the provider release vote on Friday
1st August, expected to close Monday 4th August.
-
Jarek is traveling to Beijing for Community Over Code Asia
around the 4th August release date and flagged
connectivity concerns, so he
plans to move the provider release ahead of that trip
rather than leave it
pending.
-
Jarek noted Shahar's pending ad hoc hook provider release will
be folded into this same cycle instead of shipped separately.
-
Jarek asked that provider PRs be reviewed ahead of 3.3.1 PRs,
especially any with dependencies on 3.3.1 fixes.
-
Pandas 3 Compatibility (Rahul Vats)
-
Rahul raised this during his 3.3.1 update, flagging an open
question on one of Jarek's PRs and asking whether it might affect
client-side deserialization.
-
Elad asked for context for the group, and Jarek explained the
background: Pandas 3 has been installable alongside Airflow
since January
2026, and XCom data serialized under Pandas 2 fails to
deserialize under
Pandas 3 because Pandas 3 removed an import Airflow's
built-in Pandas XCom
serializer depends on. Jarek's PR would auto-convert Pandas 2
serialized
DataFrames to Pandas 3 on deserialization. Jarek flagged this
is a one-way
migration once XComs are converted.
-
Elad said this sounds risky, and Vikram agreed, calling it a very
significant limitation.
-
Jarek argued the risk cuts both ways: if the team does nothing,
users will eventually upgrade to Pandas 3 on their own and
hit the same
failures without warning, so shipping the conversion may be
the less risky
option. Amogh clarified the deserialization Jarek was
concerned about on
the server side is likely just stringification for UI display
purposes, not
true serialization, though Jarek noted this needs to be
checked closely
since Pandas handling is deeply embedded in Airflow Core.
-
Elad floated dedicating a full feature release, ignoring the usual
numbering, solely to this change. Vikram was hesitant, not
opposed, given
how few users upgrade version by version, and wanted to see
other options
first.
-
Elad asked whether users should be warned regardless of the final
decision. Vikram agreed this was reasonable.
-
Decision for 3.3.1: defer the auto-conversion fix, and instead add
a pandas < 3 constraint plus a migration note warning users not to
upgrade to Pandas 3 on unconstrained installs.
-
Full Pandas 3 compatibility with auto-conversion is deferred to
3.4, which Jarek noted will likely set pandas >= 3 given it lands
roughly 9 months after the Pandas 3 release.
-
Elad asked that a safeguard be added to prevent constraints from
ever rolling back from Pandas 3 to Pandas 2.
-
Amogh asked whether there's enough incentive for users to actually
upgrade to Pandas 3, so the minor release isn't wasted effort.
-
Elad asked whether the change has implications for custom XCom
backends. Jarek confirmed it's specific to Pandas and doesn't
affect custom
backends.
-
Road to Common AI 1.0 (Vikram Koka, Kaxil Naik)
-
This came up as Rahul and Jarek wrapped their discussion of the
provider release cycle, when Vikram mentioned he had held up
the Common AI
provider 0.7 release pending a PR that has since been
reverted, and the
provider is now ready to release.
-
Kaxil explained the reverted PR had proposed exposing all Google
and AWS APIs to AI agents via hard-coded API lists. This was rejected
because agents work better against CLIs or MCPs than raw APIs.
-
Both AWS and Google now ship native MCP servers, and Kaxil noted
Airflow's Common AI provider already supports both MCP
toolsets and agent
skill toolsets.
-
Agent skill toolsets support local files as well as remote,
GitHub-hosted skills via the Git provider.
-
Kaxil noted the AWS MCP server allows scoped access, for example
restricting deletion operations, which provides a governance layer.
-
Jarek asked whether marketplace installs are supported. Kaxil
clarified there's no native connection to an HTTP marketplace
endpoint, but
installing a skill from a GitHub URL sourced from a
marketplace works today.
-
Full discussion of the Common AI 1.0 roadmap was deferred to the
next call.
-
Kafka Event Producer, formerly Event State Listener (Christos Bisias)
-
Vikram introduced this as a follow-up from last call's catch-up on
the event state listener, noting it took a while to get
Christos's feedback
on the original PR.
-
Christos reported on the two to-dos from last call: renaming the
plugin and updating the documentation with clear use cases.
-
Christos renamed the feature from "Kafka listener" to "Kafka event
producer" for clearer framing, since it's a listener to Airflow but a
producer to Kafka, and the old name was causing confusion. Shahar and
Vincent reviewed quickly, and the PR merged a couple of days
ago, in time
for the provider release.
-
Christos updated the documentation with common use cases:
monitoring the state of multiple DAGs and task instances
externally without
querying the database, coordinating multiple DAGs together,
and having an
external pipeline listen for a DAG to finish, or trigger one,
in either
direction.
-
Elad asked how this compares to the asset watcher and whether a
user considering both features would know which to choose, and whether
there's a single place in the docs that explains both
together. Christos
confirmed no unified doc exists yet. Elad said this is
something the team
should build, since DAG authors look for one unified solution
to a problem
even if implementation details live in separate places, and
offered to work
with Christos on it offline.
-
Jarek clarified the core distinction: an asset watcher subscribes
at the DAG level, while the Kafka event producer can operate
at the level
of an individual task within a DAG. Elad suggested this
distinction, in a
couple of sentences, is enough to add to the core docs.
-
Vikram raised a persona gap: DAG authors typically don't have
access to Airflow plugin configuration, so he asked what
coordination is
needed between personas in that situation.
-
Christos asked whether the explanation belongs in a section under
the Kafka event producer docs or somewhere more central
alongside other
related features, such as the consumer hook, asset watcher,
or a sensor.
Elad reframed this as two separate levels: documentation
clarity, and a
deeper concern that requiring a DAG author to touch the
plugin mechanism
weakens the feature itself, since many DAG authors don't have
plugin access
and have to coordinate with a devops team. Elad's position is that
DAG-author-facing features should avoid the plugin mechanism where
possible, citing timetables as a cautionary example of a plugin-gated
feature with low adoption.
-
Christos explained the feature is enabled via configuration and
applies filters so DAG authors aren't overwhelmed with
messages, and that
this uses Airflow's listener interface, which is integrated into the
scheduler and workers and adds overhead, so deliberate opt-in is
intentional rather than an oversight.
-
Vikram said Elad's feedback matches what he sees across most
organizations, including likely MWAA and GCC, and that
articulating the
persona difference matters since adoption is difficult if a DAG author
needs access to Airflow configuration. Elad reframed the
question again:
given the feature's current requirement, is there a way to
achieve the same
functionality without using the plugin mechanism.
-
Vikram asked Elad and Christos to take the remaining discussion
offline, noting he wanted to preserve time on the agenda for Dilnaz's
AIP-85 presentation. Christos agreed to follow up with Elad separately.
-
Data Quality Provider (Vikram Koka)
-
Vikram quickly noted he and Pavan have met twice since the last
call and are still working through the design. The core open question
mirrors the Kafka persona discussion: what data quality needs differ
between the DAG author, i.e. the pipeline owner, and the
business user who
owns the underlying data. Vikram noted this includes deciding
which data
quality failures should block a pipeline entirely versus flag specific
content as bad without failing the run.
-
Still open: whether this warrants a new, separate provider or
should be incorporated into existing providers. Vikram and Pavan will
return with more detail on a future call.
-
Other Development Update
-
AIP-85: DAG Importer (Dilnaz Amanzholova)
-
Dilnaz presented an abstract, pluggable interface for supporting
non-Python DAG formats such as YAML and Java, built around four key
methods: import_file, can_handle, list_dag_files, and
get_source_code.
-
This AIP was split off from a larger original proposal because of
scope, and AIP-92 is a prerequisite. Tatiana's earlier review
comments have
been addressed and the AIP is considered in good shape.
-
Pierre raised a question on the get_source_code method, noting
TP's discussion thread on AIP-108 argued against displaying source for
non-Python task SDK languages, citing complications with
external libraries
and compiled bytecode with no accessible source file.
-
Dilnaz clarified AIP-85 is an internal developer interface only,
with no end-user-visible change until a concrete importer,
such as YAML or
Java, is implemented on top of it. AIP-108 has been narrowed
to tasks only
and doesn't cover Java DAG source display, so a further AIP is likely
needed for Java DAGs to work end-to-end.
-
Rafal, Kaxil, and Dilnaz aligned on YAML-based DAGs as the first
concrete end-user deliverable. Kaxil and Pavan offered to
help on the YAML
implementation.
-
Vikram asked that the team define the full scope of work needed
for YAML DAGs to function end-to-end, including any known
unknowns, so the
effort clearly lands on end-user benefit rather than staying purely an
internal interface change.
-
Dilnaz is reading AIP-108 and syncing with TP as a follow-up.
-
Additional points raised:
-
Import errors should surface from the YAML file itself, not as
Python tracebacks.
-
The DAG bundle-to-importer mapping is one-to-many, with the
default bundle using the default importer.
-
Kaxil noted static YAML DAGs shouldn't need reparsing unless
the underlying file changes, and linked this to AIP-95,
the async DAG
processor, as a likely prerequisite. This can be a
follow-up rather than
part of AIP-85 itself.
-
AIP-115: On-Demand Task Sections was not reached due to time
constraints.
--
Vikram Koka
Chief Strategy Officer
Email: [email protected]
<https://www.astronomer.io/>
