Hey everyone, I updated our meeting notes document in the Airflow wiki to capture the notes from our dev call yesterday. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.19>
Thank you all for the amazing work. It was fantastic to see the AIPs getting wrapped up and all the different elements coming together. It was great to see the responsiveness to the feedback from the initial alphas, specifically with respect to the UI, where the UX dramatically improved from Alpha1. Loved Brent's demo showcasing the work done by the UI team. To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is on the 6th of March at the same time. Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-ProposedAgenda.3> . Best regards, Vikram -- Below is the summary from the call on Thursday: - Follow-up on action items from the last call - Development Updates and Presentations: - Update on AIP-72 Task Execution Interface aka Task SDK <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK> (Ash Berlin-Taylor and Amogh Desai) - Ash walked through a demo of the Triggerer support with the new TaskSDK. He shared how this was largely in place, with the exception of connection access. - Ash mentioned that the Triggerer is now split into two processes, using the supervisor pattern from the Task SDK. - Amogh gave an impromptu demo of the Kubernetes Executor now working with the TaskSDK. - Update on AIP-66 DAG Bundles & Parsing <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816356> (Daniel Standish) - Daniel shared a recording of the demo covering the stale DAG bundle cleanup process. - Since Workers now pull code from remote sources to run tasks and since this is stored locally to reduce latency, there is cleanup required to avoid these piling up on the worker nodes. - He said that "old / stale" could be defined by time or number of older versions. - Test plan update (Rahul Vats) - Rahul covered the testing status as documented in the Airflow 3 Test Plan <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Test+Plan> - Rahul covered the testing status of Alpha 3 and the confidence levels of the different areas including the UI and CLI. He mentioned that these are still in progress, since development is still ongoing. - He mentioned that they have started testing with focus on specific AIPs and are now reporting status by AIPs as well. - He asked the AIP owners to review the test cases and validate them as they make changes. And for the AIP owners to ensure that there was at least one run on each before the beta releases as well. - AIP-81 CLI Update (Bugra Ozturk) - Bugra covered the updated design for the Authentication work for the CLI. - Bugra asked for help from the committers on the call to review the PRs coming in for the CLI AIP, since some of the PRs were sitting for a while in review status. - Bugra commented that some of the authentication work would probably need to come in Airflow 3.1, rather than in 3.0. - Update on AIP-38 Modern Web Application <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application> (Brent Bovenzi) - Brent demonstrated an updated UX in the Airflow UI, with the Grid View or Graph View (based on the user preference) on the left hand of the page to give a consistent view with the modal views on the right hand side. - The updated UX drew a very positive response from the team on the call. - Brent showed how the new UX addressed color-blindness issues with icons and was more accessible. - Discussion topics: - Python 3.13 (not only) -driven dependency issues (Jarek Potiuk) - Jarek highlighted these three issues with 3.13. - Pendulum: He said that the Pendulum maintainer got back to him and we don't have to do anything in this area. - google-re2 → can we replace "root" usage in UI: Jarek said that Google-re2 issue is a bit more pervasive in Airflow and that after speaking with Bugra is solvable in the CLI. He was concerned about its use in the UI and API. Brent and Pierre said that this was no longer being used, so it is not needed anymore. - FAB (and sqlalchemy and pandas) → to FAB or not to FAB: The dependency conflict between SqlAlchemy, Pandas and FAB requires us to probably take out FAB as quickly as possible from Core Airflow. - Removing the old UI (Jed Cunningham) - Jed said that he is targeting next week to remove the old UI, before we ship Beta 1. - Jed also proposed moving that branch to the main repo for easier collaboration from the larger team and there was general agreement. - Bump minimum version for Airflow 3.0 to Python 3.10 (~6months ahead of EoL schedule) (Ash Berlin-Taylor) - Ash proposed this and asked if there were any strong feelings against it. - There was significant discussion about what this meant for Airflow 2.x vs. 3.x including the implications for Provider releases going forward. - The decision was to take this to the mailing list. Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
