Hey everyone, Thank you to everyone who made the dev call yesterday. I updated our meeting notes document in the Airflow wiki to capture the notes. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.32>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for the 31st of July at the same time. The agenda is pretty open for now and is focused on development updates for the AIPs targeted towards Airflow 3.1. Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-ProposedAgenda.13> . Best regards, Vikram -- Below is the summary from the call: - Catch-up on action items from last call: - None - Development Updates - UI Plugins Update (Pierre and Brent) - Pierre shared that the React plugins implementation is nearly complete, including embedded web applications now working inside the Airflow UI. - Pierre also said that key technical challenges have been resolved including react library sharing between plugins and Airflow Core. - Pierre raised one issue which had been open about where to keep the CLI/Python scripts to bootstrap React project. Based on team feedback, it was resolved to have those live within dev-tools. - Brent gave a quick demo of the React plugins working in the Airflow UI - Outstanding work on this stream before the 3.1 release includes documentation and examples to demonstrate both iFrame and React app integration, as well as CSS fine-tuning for wider screen layouts. - Task SDK Update (Kaxil, Ash, and Amogh) - Kaxil shared that multiple parallel work streams are active right now for this AIP - Ash is prototyping the common object sharing approaches (symlink vs. rendering). - Utility code migration is happening based on that to appropriate locations (task SDK vs. shared). Serialization structure changes are also being done to support this. - The Core Operator cleanup is happening, largely based on TP's work. Base Operator is now removed from Airflow models. Similar removal planned for Map operator and other core Operators. - Branch Operator bug identified and is being addressed. - Python 3.13 (Jarek) - Jarek shared that Python 3.13 is now available for all users and gave a quick demo of the same. - Jarek also said that the Flask Application Builder (FAB) dependency has now been moved and a truly FAB-less Python 3.13 deployment is possible. - Jarek also shared that Python 3.13 fixed a long-standing file description duplication bug. This bug could explain the recent "too many files open" issues in supervisor tasks. - Deadline Alerts Update (Ramit and Rajesh) - Ramit shared that he had done performance testing on different approaches using a standalone Python script on Postgres, with 10 million rows. - He tested 3 approaches: Two separate tables, view-based single table, and indexed single table. - The index approach (state + deadline time) had the best performance and therefore led to the implementation approach of adding a state column to the deadlines tables. - Ramit also shared that async callbacks would be run on the triggerer and synchronous callbacks on the workers. - Rajesh shared that the deadline detection code is nearly complete, with the Slack and SMTP notifiers being implemented as async functions. Documentation is also in progress, with an estimated dev completion in about 2 weeks. - There were questions around performance considerations and running under scale. - Discussion Topics - Airflow 3 adoption blockers - Vikram started the discussion with the commonly raised blockers that he had seen or been asked about: - Changes in imports: Vikram said that there seemed to be a significant lack of awareness of the upgrade utilities and the upgrade guide. Pavan chimed in saying that he had seen this too. There was some discussion about how to highlight this better on the docs site or in the UI, without a clear resolution. - Missing UI elements such as UI filters: Brent chimed in saying that these have been identified as Github issues and are in the process of getting resolved. Brent also asked for community help here. - Database size causing long migration times, due to long database upgrade times: Again, something covered in the upgrade guide, with the recommendation to clean up the database before upgrade. - Vikram also said that he has been using the "affected_version:3.0" issue tag to track adoption issues and then with priority. - The ask for the community is to "self-assign" adoption issues before the next meeting. - Action Items - Pierre: Continue React plugin dev tools development - Brent: React package requirements optimization after vacation - Jarek: Investigate file description issue workarounds for pre-3.13 - Dennis and Ramit: Finalize deadline alerts implementation - Kaxil, et al: Continue TaskSDK / shared code changes - Vikram: Clean up the Airflow 3.1 wiki structure and add Augusto Hidalgo to discuss Benchmark updates to next dev call. - Community: Create, tag, and self-assign (if possible) Airflow 3 adoption issues before next dev call Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
