Hey everyone, Thank you for attending the dev call on the 26th of March. I updated our meeting notes on the Airflow wiki and the link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.34>
To everyone who attended the meeting, please check the summary and add anything I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for the 9th of April at the same time. Please note that due to the US daylight saving time change, this time may be off by an hour for your time zone. It is scheduled for 8 a.m. Pacific Time on 9th of April. Since most of us have been working through release activities for Airflow 3.2.0 and associated providers, the agenda is relatively light. If you would like to discuss a particular topic, please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.36> . Best regards, Vikram -- Below is the summary from the last call: - Catch-up on action items from last call - None - Airflow 3.2 Development Updates - Testing / Release Manager Update (Rahul Vats) - Rahul gave a 3.2 release and testing update. The current target is to cut RC1 on Monday, March 30th. - Beta 2 has already been cut and testing is actively in progress. From this point onwards, Rahul will be cherry-picking bug fixes and other items tagged with the 3.2.0 milestone, rather than fast-forwarding with main. The cherry-pick strategy was shared on the dev list. - Migration testing is now in good shape. The blockers raised on the last call around migration performance have been resolved with community help, and the migration scripts have been tested across all three supported databases with no remaining issues. - Regression testing for the SDK and other areas is looking clean, and there are currently no critical or high priority issues. UI sanity testing is still in progress with a few medium priority bugs identified, but none are considered RC blockers. - Rahul specifically called out AIP-76 Asset Partitions as being in good shape, with solid testing done by Atul. Deadline Alerts owners were asked to update the test plan status, as the current one appeared to be a reuse from the 3.1 async work. Multi-team testing looks mostly complete with one open issue remaining. - Ash noted that 3.2 is now effectively in feature freeze — no new features will be added, and anything not already in will need to wait for 3.2.1 or 3.3. - Community was asked to test Beta 2 and report any high or critical priority issues on the dev list or Slack, using the beta label for tracking. - UI / API swim lane update (Pierre Jeambrun) - Pierre reported that the team has been focused on bug fixes and improvements targeted for 3.2. - Good progress has been made on the optimization front, with improvements to the dashboard page, DAG run listings, DAG listings, and log streaming. These will ship in 3.2, though more optimization work remains. - Deadline Alerts (Dennis Ferruzzi) - Dennis shared a detailed update in Slack the night before the call. - There is a PR currently open and under active review. Ash, Amogh, and Kaxil had all provided feedback. Dennis is working through Ash's design comments and expects to have a proposal within the hour. - All remaining work — including wiring for get connection, get variable, get asset, get XCom — is done locally and ready to be submitted within an hour of the blocking PR being merged. - Anish is working on a PR to unify workload handling, which is largely complete but will need a small refactor once the blocking PR lands. - Every executor needs to be updated, and volunteers have been assigned for each, with most having draft PRs already open. - Given the timing, the team agreed that the connection access piece is likely to land in 3.2.1 rather than 3.2, which can reasonably be treated as a bug fix. - Multi-team (Niko Oliveira) - UAT was largely completed last week, with just one remaining open issue related to CI. - All features scoped for 3.2 are in, and the team is holding off on any new development to avoid unintentional merges into the 3.2 branch. - Phase 2 development targeting 3.3 will start soon. Ash noted that by default nothing will get merged into 3.2 without an explicit request, so the team can start development without risk of accidentally landing 3.3 work in 3.2. - Python Async Operator docs (David Blain) - David was not on the call. Deferred to next call. - Discussion Topics - AIP-99 Progress and Demo (Pavan Kumar) - Pavan gave a live demo of AIP-99, the Common Data Access Patterns + AI work, which generated a lot of enthusiasm from the team. - For those unfamiliar, AIP-99 uses a Pythonic AI framework under the hood to enable unified, AI-assisted access to different data sources via existing Airflow hooks. The goal is to allow agents to connect to any database or data source Airflow supports, in a consistent way. - The demo covered two example DAGs. The first was a data analyst agent using a SQL toolset connected to a Postgres database with Human-in-the-loop (HITL) review enabled. The agent fetched data, responded to natural language queries, and paused for human input before proceeding — all within the task instance UI. The current implementation runs on a worker with XCom as the backend for agent/human communication. - The second demo showed a file analysis agent, using the Airflow Object Store interface to read a sample image and return a structured analysis of its key components. - Amogh suggested that when HITL review is active, there should be a clearer visual indicator in the UI so users know the task is waiting on their input. - Ash and Daniel noted that the experience feels conceptually different from a typical DAG run, though agreed that naming this distinction is hard enough that leaving it as-is for now is reasonable. - Pavan noted the work is approaching a 0.1 release, planned for the following week, and that recorded demos will be shared. - Shubham expressed excitement about the demo and the potential use cases. - Time based, scope-restricted Personal Access Tokens / PATs (Dheeraj Turaga) - Dheeraj raised the challenge that users are currently generating JWT tokens and passing them to LLMs to interact with Airflow programmatically, but those tokens are short-lived. The ask is for longer-lived tokens with admin-level controls — including the ability to revoke tokens and to scope them to read-only or limited permissions, to prevent LLMs from taking unintended write actions in production. - Ash noted that this is a meaningful ask but complicated by the Auth Manager architecture, where each auth manager has its own mechanisms and token lifecycle handling would likely need to be implemented per auth manager. - Karthikeyan highlighted the specific challenge of a user wanting read/write access for UI interactions but read-only access for LLM interactions — something tokens don't currently capture. He also shared a link to an existing token invalidation issue (#47952) and a related Claude Code issue around token refresh. - Jarek noted that token revocation has recently been added (PR #61339 merged by Pierre), and that token refresh is already possible while a token is valid. He suggested that an agent could potentially be instructed to refresh its token, though Ash noted this doesn't address the case where a token has fully expired. - Bugra suggested separating the concerns of token lifetime extension and token revocation, as they interact with different layers — token lifetime is tied to the auth manager, while revocation may be interceptable at the base auth manager level. - The general consensus was that this is a valid and important ask that requires deliberate design, particularly around security and scope controls. Jarek noted this is part of why he had reservations about MCP support — these security design questions need to be thought through carefully. - Dheeraj agreed to start an async discussion on the dev list to continue the conversation. - DualStatus Manager backwards compatibility (Christos Bisias) - This was an unplanned topic raised by Christos. A lazy consensus vote on the dev list to remove the DualStats Manager interface passed last Saturday, and Christos is working on a PR to remove it. - The question was whether it could be removed now given that it appeared in a recent provider release. - Ash clarified that since provider code referencing the DualStats Manager has already been released on PyPI, the import cannot be removed until those provider versions are no longer supported — roughly when the minimum provider version reaches Airflow 3.3, which is approximately 3 minor versions away. - The implementation can be changed (e.g. aliased to the new Stats Manager or made a no-op), but the import path must remain intact to avoid breaking released code. - Ash suggested Christos check whether the existing provider usages are wrapped in a try/except ImportError — if so, removal may be possible sooner. If they rely on a version check instead, the import must stay. - Daniel suggested that aliasing DualStats Manager to Stats Manager would be the simplest path forward, since the underlying behavior has already been moved. - Ash closed by reminding everyone to test Beta 2, with RC1 targeted for Monday. He also noted that 3.3 is planned to be a shorter cycle of roughly 3 months. Vikram Koka Chief Strategy Officer Email: [email protected] <https://www.astronomer.io/>
