Hey everyone, Thank you for attending the dev call yesterday. I updated our meeting notes document in the Airflow 3.x wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.x>to capture the notes. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.10>
The meeting was focussed primarily on user feedback regarding Airflow 3 and how to solve all adoption issues. We also agreed to focus on these issues for the next month. More details below. I have also updated the Airflow 3.x wiki page <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.x> with a specific "Airflow 3 adoption focus" section. To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for the 6th of November at the same time. Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.11> . Best regards, Vikram -- Below is the summary from the call: - Development Updates and Presentations: - Airflow 3.1.1 patch release update (Kaxil Naik) - Kaxil outlined that the 3.1.1 RC2 was cut earlier in the day and he expected 3.1.1 to be generally available on Monday next week. - Kaxil also said that he planned to cut the 3.1.2 RC1 before the end of October. Kaxil will be taking leave for a few weeks after that and Ephraim will be cutting the following patch releases. - Airflow 2.11.1 patch release update (Jarek Potiuk) - Jarek spoke that this was primarily driven by a need to fix a key CVE and that he had to change a key dependency, but the tests were now working - Jarek said this would be a combined Airflow 2.11.1 and Fab provider 1.5.4 release - RC1 is planned for community testing, since this addresses multiple CVEs in dependencies. - Jens offered to help with testing support. - Discussion topics - Feedback from users about project at Airflow Summit - Vikram asked the team to share the feedback they had heard from the Summit regarding any Airflow 3 adoption challenges or critical issues - Amogh went first saying that he was folks had raised issues with niche use cases - primarily when they were leveraging the Airflow meta database directly for temporary data storage and therefore the lack of direct DB access was a blocker in Airflow 3 adoption. - Vikram spoke about how we had told users in the past that they could work around this by explicitly defining a database connection to the Postgres database and using it explicitly, rather than implicitly through the Airflow internal interfaces, but also added that he wasn't sure if it was documented anywhere. Amogh said that he wasn't aware of that expressed solution and would create a Github issue to make sure it was documented. There was a larger discussion around this, including that users could leverage the published APIs (going through the API server) for this capability now. - Dheeraj spoke about the issues his team at Qualcomm had faced in their upgrade. This was following up on his earlier email to the dev list and said that they had faced the issue above as well and had worked around it as expressed above. - Dheeraj also raised the issue that the import changes were annoying, and that the Qualcomm team was not aware of the Ruff based utilities for upgrades. There was significant discussion around this as well and if this was a point in time issue with the early 3.0 builds. Ash asked Dheeraj if he could retry this with Airflow 3.1, since there was so much compatibility work done to keep the imports working and Dheeraj agreed to get back on this. - Dheeraj also said that there were several UI challenges around performance and a couple around navigation. Specifically, slow to load large DAGs, to list lots of DAGs, and log scrolling for large files. Also, viewing the UI from a mobile phone was problematic in 3 as compared to 2.x. Brent and Pierre said that there were performance improvements in 3.1.1, but work was in flight around this. - Dheeraj also said that very large DAG files i.e. with 23,000+ tasks were choking the DAG file processor. Kaxil asked Dheeraj to open a Github issue around this. - Devlist followup on focusing on adoption issues - Vikram asked the team if there were any objections to the proposal shared on the devlist to devote the next month i.e. mid-Oct to mid-Nov towards focusing on Airflow 3 adoption issues instead of significant new development. There were no objections to this. - Vikram then summarized that he would add this focus area to the Airflow wiki and that we could release a set of patch releases over the next four weeks to address these issues. - Vikram also said that the logical timing for Airflow 3.2 would be to start development in earnest in mid-Nov, with a target release of March 2026, accounting for the holidays in late November and late December. There was general consensus. - Why more providers needed ? Jarek Potiuk <https://cwiki.apache.org/confluence/display/~potiuk> - deferred because of lack of time -- Vikram Koka Chief Strategy Officer Email: [email protected] <https://www.astronomer.io/>
