Hey everyone, Thank you to everyone who made the dev call yesterday. I updated our meeting notes document in the Airflow 3.x wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.x>to capture the notes. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.7>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for the 28th of August at the same time. The agenda is focused on development and test updates for Airflow 3.1. Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.8> . Best regards, Vikram -- Below is the summary from the call: - Catch-up on action items from last call: None - Development updates - AIP-72 Task Execution Interface aka Task SDK <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK> update (Kaxil Naik) - Kaxil shared TP had eliminated the Base Operator completely from airflow.models, and that the serialized base / mapped operators were being used exclusively - Kaxil also said that the DAG object removal from models is in progress and that he moved to have the big DAG serialization PR out this week, so that all the Task information would be in the serialized DAG version without relying on the shared code. - This would then enable the server-client separation and independent deployment upgrades - He said that Ephraim was migrating the airflow exceptions code to appropriate SDK/Core locations - The one blocker for full separation was the shared config parse module. - Benchmark (AIP-59 Performance tests framework <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework>) update (Michal Modras) - Michal shared the initial benchmark data comparing Airflow 2.10.5 vs. 3.0.0 <https://docs.google.com/document/d/1_vgDD0_5o8plvxT-bBKPKydncEywe_vcSavsh2-9ZYA/> which was run on a GKE cluster with 4 nodes. - Initial performance results for 2 baseline scenarios were positive with Airflow 3.x being significantly faster than Airflow 2.10 - Michal said that the team would be testing with more scenarios and would publish the testing framework before Airflow Summit - The team was positively surprised by the results and there was significant interest in leveraging the framework as part of RCs and as part of pre-merge checks for complex PRs such as pessimistic vs. optimistic scheduling. - AIP-86 Deadline Alerts <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488182> update (Dennis, Ramit) - Deferred to next dev call - Airflow 3.1 test plan update (Rahul) - Rahul shared the updated 3.1 test plan <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.1+Test+Plan> and covered the thinking behind it. - He explained that the key focus areas were around upgrade / downgrade testing, Human in the loop (since the development there was nearly complete), and the Task SDK decoupling. - Rahul also said that the Task SDK decoupling was of critical priority and needed focus. - Jarek asked about testing of internationalization, which led to the discussion below. - Discussion topics - UI Internationalization release process - Following up the 3.1 release testing plan, the team discussed the key steps to releasing the UI internationalization work. - Jens spoke about the earlier discussed policy for this which included a reminder to the dev list about 2 weeks before RC, which would serve as a notice to the language owners to make sure that the translations are up to date before the release. - Shahar volunteered to coordinate between the release manager (Kaxil) and the language owners for the 3.1 release. - Airflow 3 adoption blockers (issues with label priority:upgrade_to_airflow3 <https://github.com/apache/airflow/issues?q=state%3Aopen%20label%3A%22priority%3Aupgrade_to_airflow3%22> ) - There was a good discussion of all the open (around a dozen) issues which were tagged as Airflow 3 upgrade blockers. - Action items - Shahar: Sync up with Kaxil and send out note to language owners about translation updates and validation - Rahul: Update test plan and raise issues with contributors as needed - Michal: Share benchmark framework when ready - Vikram: Ping Rajesh, Dennis, et al. about sync SLA callbacks Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
