Hey everyone, Thank you for attending the dev call on Jan 29th. I updated our meeting notes on the Airflow wiki and the link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.30>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for the day after tomorrow the 12th of February at the same time. The agenda is already populated, primarily with swim lane updates and Airflow 3.2 AIP updates. If you would like to keep this call to discuss a particular topic, please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.32> . Best regards, Vikram -- Below is the summary from the call: - Swim lane updates: - UI Test framework (Rahul Vats): - Rahul shared that we now have 65 test cases with good input from the community. - He also shared that as a result CI now takes 15 minutes for these tests and optimization is needed, with one approach to move some of these into unit tests. Rahul said he is working with Brent on this. - Rahul also mentioned that this has already been valuable in catching a bug in main before the 3.1.7 release. - Still working on an important issue with Grid performance with large DAGs. Kaxil highlighted a great PR from Christos which fixed a performance issue i.e. #54103 - UI / API (Pierre Jeambrun): - Pierre shared an overall update including several feature changes that will show up in Airflow 3.2 such as permissions changes for API-level access controls, UI customization changes, and React compiler optimizations. - Kartik had added a significant change in PR #60274 which resulted in noticeable improvements in API server performance, by eliminating duplicate fetching. - Airflow 3.2 development updates: - Task SDK Update (Amogh) - Amogh said that the Task SDK work is nearing completion excluding small core imports and that he had to make changes to the Deadline Alerts for SDK separation compliance which needs review from the AWS team. - He also added that he was working on a declarative connections loading scheme via YAML spec to decouple providers. - Deadline Alerts (Dennis Ferruzzi): - Dennis shared that sync callback support PR is ready for base / local executors. Niko has reviewed this and will have changes. - Dennis said that the behavior of the sync callbacks will be different from the old SLA callbacks because these will be run by the workers rather than by the DAG file processor. - Kaxil said this could have concurrency implications and asked for the documentation changes detailing this behavior. - Multi-team (Niko / Vincent): - Niko said the executor updates were in flight with the AWS Batch executor work completed and others assigned to community members. Niko said that the celery executor work is complex and still in progress. - The second stream of work was on the UI, with the pool selector UI updates by Vincent nearing completion. - Niko said that the dag_id uniqueness work was difficult and probably would not be in the 3.2 release. - Asset Partitions (Daniel Standish): - Daniel shared that the scheduler changes for DAG scheduling based on partitions was in flight, with several PRs having been added by Wei. - The next big item was backfill support for asset partitions. - Daniel also shared that the UI work was being planned with Brent. - AIP-98 Async support for Python Operator (David Blain): - David shared that this had already been merged and would be part of Airflow 3.2. - There was an open item for documentation around this, specifically regarding the use cases supported. - David shared that this would eventually lead to the Iterable Operator work in AIP-98. - Discussion topics: - AIP-95 Providers lifecycle governance (Vikram) - Vikram shared that the first PR for the updated governance process was merged and that the new process enables the sponsoring of additional providers. - Initial candidates identified include Vespa and Informatica, which are both in flight. - AIP-94 CLI Decoupling (Bugra) - Bugra noted that the voting period for this AIP was currently active through the week and invited feedback. - Bugra elaborated that the current plan was to have Airflow CLI commands proxy to Airflow CTL for remote operations, without any user-facing changes. - Local admin commands such as DB clean would remain unchanged. - Kaxil suggested an alternative i.e. single Airflow CTL set of commands for all operations. -- Vikram Koka Chief Strategy Officer Email: [email protected] <https://www.astronomer.io/>
