Hey all, I have updated our meeting notes document to summarize the discussion from our 25th July dev call for Airflow 3.0.
Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-25July2024 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree with anything in the Summary, please voice your opinion. I will send a separate email with the agenda for the next meeting on 8th August, but if someone has an item for the agenda, please reach out to me or feel free to add it to the doc <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> . Regards, Kaxil ------ Including the Summary here too (might break formatting): *Catch-up on action items from last call* - Kaxil created GitHub issues for workstream items with no owners; they have also been added as sub-items in this meta GitHub issue that tracks the 3.0 release <https://github.com/apache/airflow/issues/39593>. We had already had some interest from community members on some of these items, check individual issues for it. *Discussions* Branching Strategy: - As soon as 2.10.0rc1 is created, the Airflow main branch will become the Airflow 3 branch on *09 Aug 2024 *. - Providers will continue to be developed on the main branch since we have the framework to run provider tests against older Airflow versions. - Bugfixes: If there is a PR with a bugfix to main, the committers should ask the PR author to raise a PR against v2-10-test branch too. The team agreed to create a mailing list thread to discuss different approaches offline. The thread has been initiated here <https://lists.apache.org/thread/gjlotg36lvp6qxzblk4fwv6wymof09h1>. Which Python version should we support in Airflow 3? - The consensus was to keep things as-is i.e following the current Python-support policy <https://github.com/apache/airflow?tab=readme-ov-file#support-for-python-and-kubernetes-versions> where we would remove the support for a Python version when it is EOL. Airflow 2 support policy including scope (feature vs bug fixes + security only) & support period - We might release 2.11 and mark it as a bridge release to add compatibility shims or deprecation warnings similar to 1.10.15. But we will decide on it as and when such a need arises. - For providers, we will need to increase the support window for the minimum Airflow version from the current 6 months. We will continue Airflow 2 provider support for a year or longer, exact period TBD. We will revisit this again in the coming months. If we cut 2.11 (or whatever the final 2.x version is), then after 6-12 months we could increase the minimum Airflow to 2.11. There were ideas around using a "compat" provider that might help with AF 2 and AF 3 stuff. This will be another thing we will discuss async at a later time to keep current focus on scope & AIPs. *Workstream statuses for ones that does not have a vote started:* - AIP-57 Refactor SLA Feature (Shubham Mehta): - It was agreed that for now, we will mark it for removal in Airflow 3.0 and adding it back in 3.1. We will assess again in the coming dev calls if something changes. Elad has also offered to help if needed. - AIP-67 Multi-team deployment of Airflow components (Jarek Potiuk ): - Jarek is waiting to hear back on his answers to some of the doubts raised in the discussion thread and AIP. This might also be moved to Airflow 3.1 since it depends on AIP-72 and AIP-66 and the implementation hasn't been started on those yet. - AIP-68 Extended Plugin Interface for React Views (Jens Scheffler / Brent Bovenzi ): - As of today, the vote has been started on the mailing list (link <https://lists.apache.org/thread/x8k3khtw6nqbt88r9qlc4b0s1y4bwvf7>). During the meeting, a few things were discussed, including potentially renaming UI plugins to UI Extensions as the "Plugin" word has been overloaded. The other suggestion was to rename the AIP title to "New Interface" instead of "Extending the interface." - AIP-76 Asset Partitions (Tzu-ping Chung): - TP will send this out for VOTE in the next couple of days. - AIP-79 Replace Flask App Builder (Brent Bovenzi ): - Two big questions pending: 1. Replacement for FAB Auth manager 2. How does adding a FAB as Provider work? Can it allow users to make older plugins work? - Jed has been working on a POC on it. He is on vacation this week, but based on initial investigations, he was positive about the progress. We will check it back in the next dev call when he is back. - The pending item is also to update the AIP to add the decoupling of Providers's Connections metadata & form from FAB. - AIP-80: Explicit Template Fields in Operator Arguments (Tzu-ping Chung ): - VOTE has been kicked off here <https://lists.apache.org/thread/s5g0zvjjxlpgnp718vs6n86qtlgzcgpw>. - Poll external Datasets to have event-based DAG scheduling (Vincent BECK / Shubham Mehta): Draft AIP <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow> has been created as mentioned in this mailing list thread <https://lists.apache.org/thread/b7kl9sng7zn81dchn5nqom336ql5pnf2>. - Respect permissions in CLI: Buğra Öztürk - It now has an AIP-81: Enhanced Security in CLI via Integration of API <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-81%3A+Enhanced+Security+in+CLI+via+Integration+of+API> and is awaiting feedback. - Returning DAG Results / Synchronous DAG Execution <https://docs.google.com/document/d/1CXis3Ovmxm3pt4moiBIpsajX7GmRpGbcEfwt_blkSLI/edit?usp=sharing> : Vikram Koka - The first proposal (Make Execution DAG non-unique) from the original doc is separated out into a different workstream led by TP. - Language support has been excluded from the scope for Airflow 3.0. For now, the only change considered would be to add the support to the existing Trigger API to return an async result which doesn't need an AIP. - The other changes for GenAI will now be implemented as a new provider. - Make Execution Date to be non-unique for a DAG: Tzu-ping Chung - In the next couple of days, a discussion thread with a draft AIP will be created by Tzu-ping Chung , followed by a vote. - Splitting providers from the mono-repo (Kaxil Naik) - Kaxil is not going to propose this anymore as he agrees that doing this now will increase the work and split the focus of folks. So we are going to NOT split providers from mono-repo to laser focus on Airflow 3 feature work. - We might think about it at a later date after Airflow 3.0 is released. - AIP-49 OpenTelemetry Support for Apache Airflow <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow> : - The PR for the 2nd part <https://github.com/apache/airflow/pulls?q=is%3Apr+opentelemetry+is%3Aclosed> got merged for 2.10 and the AIP is marked complete. - Howard Yoo 's comment: Initially we thought to collect logs in standard way (scraping from log files), but found a much better way to collect logs, by utilizing log api to get the task logs, and then attaching them as 'span events' which would be much more useful. The extraction of Task Logs and attaching them as span events is thus included in the AIP-49, thus it is not 'excluded', albeit not as a separate 'log' data as originally thought. - StatsD vs Open Telemetry - There was a discussion on whether we should mark StatsD as removed in Airflow 2.10 itself in the docs here <https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html>. The conclusion was that we should discuss this more as this has a wider impact including but not limited to the official Helm Chart which uses StatsD, the services providers (Astronomer, Composer, MWAA) and other long-time Airflow users. OTel also had limitation around character length limit and that you have to use tags with OTel but it didn't allow aggregating metrics with different tags etc. Shubham Mehta is going to check this out with his team and see if there are other concerns. Triage Help needed: There are 360 open feature requests <https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Akind%3Afeature+sort%3Aupdated-asc>. Some we might want to consider and seek help for in 3.0, and others we should close as not planned. Elad, Jens, Jarek, Vikram & Kaxil have volunteered to look at those issues. Elad is also going to check with Dosu team to see if we can re-label these old issues so they are more accurate. Categorizing these issues into "areas" label was one of the next steps too so that the volunteers can look at a particular area or some segment to divide & conquer. Elad Kalif could you take the one on for issue-triage?
