Thanks for the meeting notes Kaxil, this looks right. One key takeaway from the meeting was the relationship between AIP-38, AIP-68, and now AIP-79. The sooner we can clarify the dependencies here and establish the fallbacks i.e. with respect to FAB auth, the better from a risk management perspective. I am glad to see it noted here, but I believe we need to track it in every dev call until resolved.
On Tue, Jul 16, 2024 at 4:18 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > Hey all, > > I have updated our meeting notes document to summarize the discussion > from our 11th > July dev call for Airflow 3.0. > > Link: > > https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-11July2024 > > To all those who attended, can you please double-check and add if > I have missed anything? > > To all those who didn't join, if you disagree with anything in the Summary, > please voice your opinion. > > I will send a separate email for the agenda for the next meeting on > 25th July but if someone has an item for the agenda, please reach out to me > or feel free to add it to the doc > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda > > > . > > Regards, > Kaxil > > ------ > > Including the Summary here too (might break formatting): > > *Catch-up on action items from last call* > > - AIP-61 (Hybrid Executor): On track for Airflow 2.10. > - Kaxil Naik deferred Creating GitHub issues for items in the Workstream > table with no owners to next week. > > > *Check-in: Workstreams & workstream owners (Airflow 3 Workstreams > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams > >)* > > - AIP-68 (Extending plugin interface): This is finalized for 3.0 instead > of 2.10 > - AIP-38 (UI Modernization): Brent Bovenzi presented the progress of > AIP-38 in the Airflow 2.3, 2.4, 2.7 & 2.8. An update on what's pending > has > been added to the AIP-38 page. > - A new AIP was split from AIP-38 to replace FAB: AIP-79 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79+Replace+Flask+App+Builder?src=contextnavpagetreemode > > > . > - The new Plugin interface for UI will be defined by AIP-68 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels > > > - Brent Bovenzi & Jens Scheffler : Could either of you add what you > presented for AIP-68 in the AIP doc, around breaking changes to > existing > plugins & CLI to initialize a starter project? > - *Risk / Action Item*: There were discussions around the impact on > Auth due to the removal of FAB. KeyCloak Auth Manager was mentioned > as a > potential replacement for Enterprise installation and using FAB > Airflow > provider for local development. Fab auth manager via FAB > provider might be > a fallback until we get the Keycloak implementation. We should > test this to > make sure that it doesn’t become a blocker and that FAB provider is a > reliable fallback. This should be noted in AIP-79 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79+Replace+Flask+App+Builder?src=contextnavpagetreemode > > > . > - AIP-67 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode > > > (Multi-team): Jarek > Potiuk presented the revised AIP. The AIP has been updated, and Jarek > has > answered the questions that were asked in the chat in this mailing list > thread < > https://lists.apache.org/thread/w6g207ws5swq09c52rm46doy4b8lp5pc> > . > - As a follow-up to the call, Jarek also created a mailing list > thread > <https://lists.apache.org/thread/b5ycrsmjw4b3tn2jpc368ck2l44vm53p> > about > whether to finish AIP-44 for AF 2 or not. > - AIP-62 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-62+Getting+Lineage+from+Hook+Instrumentation > >(Lineage > from Hooks): Maciej Obuchowski presented current progress and is > targeting > Airflow 2.10 for its completion from the Airflow core side, the rest > will > be in Providers. > - AIP-57 Refactor SLA Feature > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature > >: > Kaxil & Shubham both heard back from Sungwon Yun. Unfortunately, he > doesn't > have the capacity to work on it, but he is going to meet Shubham Mehta > and > the team for knowledge transfer. > - AIP-65 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-65%3A+Improve+DAG+history+in+UI?src=contextnavpagetreemode > > > (DAG > Versioning—UI side): Jedidiah Cunningham decided to keep this AIP as is > instead of making it part of the UI Modernization AIP. > - AIP-66 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > (DAG > Versioning—Execution side): Jedidiah Cunningham marked it as ready for > review. The email thread is here > <https://lists.apache.org/thread/l8ksl144xd43jfk1wk3kz77t1xgbbq7z>. > - AIP-69 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor?src=contextnavpagetreemode > > > (Remote > Executor): Jens Scheffler wants to start voting, but it is dependent on > AIP-44, which is discussed separately here > <https://lists.apache.org/thread/b5ycrsmjw4b3tn2jpc368ck2l44vm53p>. > - AIP-72 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > (Task > Execution SDK): VOTE > <https://lists.apache.org/thread/hcdb8tb2l09kjho5mtw4gy3mk31g84ox> has > succeeded as of July 16. > - AIP-73 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-73+Expanded+Data+Awareness > > > (Data > Awareness): Tzu-ping Chung & Constance Martineau plan to remove AIP-74 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-74+Introducing+Data+Assets > > > (Introducing > Data Assets) & AIP-75 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-75+New+Asset-Centric+Syntax > > > (New > Asset-Centric Syntax) from the draft in a week and timelines for the > rest > to follow. > - AIP-78 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-78+Scheduler-managed+backfill > > > (Scheduler-managed > backfill) was published by Daniel Standish and is under-going > discussion. > - "*other candidates for 3.0*" list: > - *Poll external Datasets to have event-based DAG scheduling: * – > This will become an AIP, Vincent BECK / Shubham Mehta working on > the draft > AIP. > - *Respect permissions in CLI*: Buğra Öztürk is working on the first > draft, timelines TBD. > > > *Closing Notes* > The group collectively agreed that all the big AIPs targeting Airflow 3.0 > should at least have the vote started before August 1. Anything after that > does not have a realistic chance of completion. This is to not rush the > AIPs but to prioritize the key items, especially the ones that contain > breaking changes, get them vetted out properly, and make them better for > the initial MVP launch. >
