Hey all, I have updated our meeting notes document to summarize the discussion from our 11th July dev call for Airflow 3.0.
Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-11July2024 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree with anything in the Summary, please voice your opinion. I will send a separate email for the agenda for the next meeting on 25th July but if someone has an item for the agenda, please reach out to me or feel free to add it to the doc <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> . Regards, Kaxil ------ Including the Summary here too (might break formatting): *Catch-up on action items from last call* - AIP-61 (Hybrid Executor): On track for Airflow 2.10. - Kaxil Naik deferred Creating GitHub issues for items in the Workstream table with no owners to next week. *Check-in: Workstreams & workstream owners (Airflow 3 Workstreams <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams>)* - AIP-68 (Extending plugin interface): This is finalized for 3.0 instead of 2.10 - AIP-38 (UI Modernization): Brent Bovenzi presented the progress of AIP-38 in the Airflow 2.3, 2.4, 2.7 & 2.8. An update on what's pending has been added to the AIP-38 page. - A new AIP was split from AIP-38 to replace FAB: AIP-79 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79+Replace+Flask+App+Builder?src=contextnavpagetreemode> . - The new Plugin interface for UI will be defined by AIP-68 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels> - Brent Bovenzi & Jens Scheffler : Could either of you add what you presented for AIP-68 in the AIP doc, around breaking changes to existing plugins & CLI to initialize a starter project? - *Risk / Action Item*: There were discussions around the impact on Auth due to the removal of FAB. KeyCloak Auth Manager was mentioned as a potential replacement for Enterprise installation and using FAB Airflow provider for local development. Fab auth manager via FAB provider might be a fallback until we get the Keycloak implementation. We should test this to make sure that it doesn’t become a blocker and that FAB provider is a reliable fallback. This should be noted in AIP-79 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79+Replace+Flask+App+Builder?src=contextnavpagetreemode> . - AIP-67 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode> (Multi-team): Jarek Potiuk presented the revised AIP. The AIP has been updated, and Jarek has answered the questions that were asked in the chat in this mailing list thread <https://lists.apache.org/thread/w6g207ws5swq09c52rm46doy4b8lp5pc> . - As a follow-up to the call, Jarek also created a mailing list thread <https://lists.apache.org/thread/b5ycrsmjw4b3tn2jpc368ck2l44vm53p> about whether to finish AIP-44 for AF 2 or not. - AIP-62 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-62+Getting+Lineage+from+Hook+Instrumentation>(Lineage from Hooks): Maciej Obuchowski presented current progress and is targeting Airflow 2.10 for its completion from the Airflow core side, the rest will be in Providers. - AIP-57 Refactor SLA Feature <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature>: Kaxil & Shubham both heard back from Sungwon Yun. Unfortunately, he doesn't have the capacity to work on it, but he is going to meet Shubham Mehta and the team for knowledge transfer. - AIP-65 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-65%3A+Improve+DAG+history+in+UI?src=contextnavpagetreemode> (DAG Versioning—UI side): Jedidiah Cunningham decided to keep this AIP as is instead of making it part of the UI Modernization AIP. - AIP-66 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode> (DAG Versioning—Execution side): Jedidiah Cunningham marked it as ready for review. The email thread is here <https://lists.apache.org/thread/l8ksl144xd43jfk1wk3kz77t1xgbbq7z>. - AIP-69 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor?src=contextnavpagetreemode> (Remote Executor): Jens Scheffler wants to start voting, but it is dependent on AIP-44, which is discussed separately here <https://lists.apache.org/thread/b5ycrsmjw4b3tn2jpc368ck2l44vm53p>. - AIP-72 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode> (Task Execution SDK): VOTE <https://lists.apache.org/thread/hcdb8tb2l09kjho5mtw4gy3mk31g84ox> has succeeded as of July 16. - AIP-73 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-73+Expanded+Data+Awareness> (Data Awareness): Tzu-ping Chung & Constance Martineau plan to remove AIP-74 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-74+Introducing+Data+Assets> (Introducing Data Assets) & AIP-75 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-75+New+Asset-Centric+Syntax> (New Asset-Centric Syntax) from the draft in a week and timelines for the rest to follow. - AIP-78 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-78+Scheduler-managed+backfill> (Scheduler-managed backfill) was published by Daniel Standish and is under-going discussion. - "*other candidates for 3.0*" list: - *Poll external Datasets to have event-based DAG scheduling: * – This will become an AIP, Vincent BECK / Shubham Mehta working on the draft AIP. - *Respect permissions in CLI*: Buğra Öztürk is working on the first draft, timelines TBD. *Closing Notes* The group collectively agreed that all the big AIPs targeting Airflow 3.0 should at least have the vote started before August 1. Anything after that does not have a realistic chance of completion. This is to not rush the AIPs but to prioritize the key items, especially the ones that contain breaking changes, get them vetted out properly, and make them better for the initial MVP launch.
