Hey everyone, I updated our meeting notes document in the Airflow wiki to capture the notes from our dev call yesterday, the 21st of November. The link for those notes is here <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-Summary.13>
Thank you everyone for all the hard work. It was truly energizing to see the demos of the AIPs coming together! To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. There's already an initial agenda for our next dev call, which is scheduled for 5th Dec. If you would like something to be added to the proposed agenda for that meeting, please add it here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-(Proposed)Agenda.3> or let me know. Best regards and talk to you all soon, Vikram -- Below is the summary from the call yesterday: - Follow-up on action items from the last call: - Update on the POC of Flask/FAB app into the Fast API (Jed Cunningham): - - The POC worked and that means we will not need a separate webserver to maintain backward compatibility with existing plugins. - In addition, this means that we don't need the new UI (AIP-38) to be far enough along to remove the old UI. The old UI can stay as-is, and support for plugins can be added in the new FastAPI webserver. - Taking this from POC to production level reality will take some work. This will be planned by Jed and Vincent. Anticipate seeing a Github project board with the items to be done in time for the next dev call. - Breaking changes update (Vikram Koka): - Vikram walked the group through the draft of the Airflow 3 breaking changes document <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+breaking+changes> . - Vikram mentioned that this is about 50% populated right now, with 50% of current known changes still to be populated. - To reduce the manual effort going forward, we will have a newsfragment template to be filled out as part of the PR process. - Development updates and presentations: - AIP-72 update (Ash and Kaxil): - Ash demonstrated the working state of AIP-72 with respect to the separation of the Task Execution code into the TaskSDK vs. Airflow Core. This was in the form of tasks created with the new Task SDK and being run on the Local Executor, using a supervisory process. The demo included the code showing how the supervisory process launched the task process and the socket pair / dup mechanism for logging. Ash also mentioned that the logging providers may need changes to accommodate the new system. The team also discussed how this would make it a lot easier to intelligently display logs in the UI, without the UI needing to reparse the logs. It was very encouraging to see this coming together. - AIP-66 update (Jed): - Jed shared a quick update regarding the progress being made on AIP-66 including the focus on the refreshing bundles from an external source. Jed also mentioned that he would be ready to open a PR for this early next week and that the demo could happen in the next dev call. He also expressed hope that the connection form work could be leveraged for the UI support here. - AIP-38 update (Brent): - Brent presented the UI progress including the new react-based DAGs list page and the dashboard page with basic statistics all leveraging the new FastAPI work. Brent mentioned that they were working on the Dag Run and task instance pages. - AIP-84 update (Pierre): - Pierre and Brent expressed gratitude for the many contributions from the community on AIP-84. Pierre confirmed that they were almost done with the migration of all public endpoints and that there were only a few remaining private endpoints to be done. - - Discussion topics: - Meta milestones (Vikram): - Vikram highlighted that from an overall standpoint, the project was generally on track with the published schedule. - He emphasized that the backend features should ideally be completed by the end of December and must be integrated by the week of the 18th of January in order to be on schedule. - He said that this was critical for the packaging and upgrade work to be done on a stable base and be completed by the end of March. He mentioned that the UI could continue to change till early March, but the Jan backend feature freeze was critical. He asked all the AIP owners to keep this in mind and be pragmatic about their AIP scope. - - Action items for next dev call: - Jed Cunningham and Vincent to confirm back regarding the Fab provider compatibility stream (Github project board). - Newsfragement template embedded within PR process (TBD, Vikram Koka to make sure it happens). - AIP owners to confirm back on completion plan for server features to be completed by mid-Jan. - PR owners to review breaking changes page for accuracy. -- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
