Hey everyone, I updated our meeting notes document in the Airflow wiki to capture the notes from our dev call earlier today. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.16>
Loved the progress on DAG Versioning. It was great to see multiple versions of the DAG and the integration of the "DAG Fetcher" concept to enable the consistent running of a defined DAG version. DAG Versioning has long been the most requested feature for Airflow and it's wonderful to see the demo of this working. Great work on this Jed, Ephraim, and Daniel Standish! It was also great to see the progress on the CLI. Thank you Bugra and team for all the work here! To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. There's an initial agenda for our next dev call, which is scheduled for 23rd Jan. If you would like something to be added to the proposed agenda for that meeting, please add it here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-ProposedAgenda> or let me know. Best regards, Vikram -- Below is the summary from the call on Thursday: -- - Follow-up on action items from the last call: - Update on the AI-83 amendment discussion (Daniel Standish and TP Chung): - The docs shared earlier were: Amendment to AIP-83 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-83+amendment+to+support+classic+Airflow+authoring+style> and Option 2 clarification <https://cwiki.apache.org/confluence/display/AIRFLOW/Option+2+clarification+doc+WIP> - Daniel and TP walked through the detailed "option 2 clarification" document above to address the questions under discussion - At a high level, there was agreement about the overall approach. There were some details to be worked through. - Elad suggested a deadline to close the discussion, so that this could be done in time for release. Vikram agreed and recommended that Daniel / TP start an email voting thread, so this could be closed by Thursday of next week. - The team also expressed appreciation to Daniel and TP for their perseverance in working through this change, since this had generated huge debate. - Development updates and presentations: - Update on AIP-66: DAG Bundles & Parsing <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816356> (Jed Cunningham): - - Jed demonstrated DAG bundles based on a Github repo, then DAG versioning and DAG fetching. - He started a DAG run based on DAG version 1, updated the DAG, and then showed how the currently running DAG continued to execute subsequent tasks based on version 1 of the DAG, rather than the latest version. - Jed highlighted that this work had been a team effort with Ephraim and Daniel Standish. - DAG versionsing is a long requested feature within Airflow and it was great to see the working demo. - Update on AIP-81 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-81+Enhanced+Security+in+CLI+via+Integration+of+API>Enhanced security of CLI by integration with API <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-81+Enhanced+Security+in+CLI+via+Integration+of+API>(Bugra Ozturk): - Bugra presented an update on the progress to date on the CLI - Bugra demonstrated the progress on making the CLI use the API for enhanced security, along with the command for connection configuration, as well as the authentication integration in flight. - This AIP has been flying under the radar for a bit, so it was a treat to see the wonderful progress here. - Discussion topics: - Doc build issues (Jarek Potiuk): - Jarek brought up the concern that the Airflow doc builds were taking 30 minutes now, from around 5 minutes a couple of years ago. He, Ash, Elad, Jed, and Kaxil all highlighted that this was a ticking time bomb. - As a quick analysis, there seemed to be three different areas which needed to be addressed here: - The doc build technology which has not been updated in years - Process changes around Airflow user documentation including retention, versioning, and search engine optimization - Doc structure changes as needed for the upcoming Airflow 3 release. - Jarek also brought up the point that the typical folks who have been patching this are all very busy with Airflow 3 development and will be till release. Constance volunteered to take on the third item identified above. - We could not finish the discussion because of time constraints, but the takeaway was to start an async discussion thread around this topic. - Airflow 2.11 release: when and what (Jens Scheffler) - Jens brought up the topic of Airflow 2.11, saying that we had discussed cutting the Airflow 2.11 release when we started working on Airflow 3 and if that was still in the plan. - The Airflow 2.11 discussion was initially triggered by the forward compatibility proposal with AIP-80. That approach had been changed before AIP approval and subsequently AIP-80 itself had been deferred to post-3.0. Since then, the Airflow 2.11 release faded away from the discussion. - Kaxil and Jens took the action item to find the issues tagged for Airflow 2.11 and to start an async discussion around the pros/cons of that release. - Action items on/before next dev call: - Daniel Standish / TP to start a voting thread for the AIP-83 amendment. - Jarek to start an async discussion around the doc build issues - Jens / Kaxil to start an async discussion around the pros/cons for the 2.11 release. --- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
