Looks cool! On Sun, Jun 30, 2024 at 6:56 PM Kaxil Naik <kaxiln...@gmail.com> wrote:
> Hey all, > > I have updated our meeting notes document to summarize the discussion > from our 27th > June dev call for Airflow 3.0. > > Link: > > https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-27June2024 > > To all those who attended, can you please double-check and add if > I have missed anything? > > To all those who didn't join, if you disagree with anything in the Summary, > please voice your opinion. > > I will send a separate email for the agenda for the next meeting on > 11th July. > > Regards, > Kaxil > > ------ > > Including the Summary here too (might break formatting): > > Catch-up on action items from last call > > - Kaxil Naik updated the AIP template > < > https://cwiki.apache.org/confluence/pages/templates2/viewpagetemplate.action?entityId=90210323&key=AIRFLOW > > > to > include the "Migration effort" section. If the template is used to > create a > new AIP doc (click "Create" → "Airflow Improvement Proposal" on the > wiki) > it should pre-populate sections. The access to editing the template > itself > is limited to PMC members & ASF Confluence Admin group. Need to > retroactively add that section in existing AIPs planned for 3.0 > [image: Screenshot of How to create a page using template] > - SLA PR <https://github.com/apache/airflow/pull/36639>: This is still > pending review. > - Marked AIP-51 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow?src=contextnavpagetreemode > > > as > completed, and the pending UI issue > <https://github.com/apache/airflow/issues/27933> will be covered in the > UI refactor. > - AIP-61: Niko Oliveira is still targeting 2.10 for completion. PRs are > raised & pending reviews: #40472 (Backfill part) > <https://github.com/apache/airflow/pull/40472> & #40017 (Scheduler > part) > <https://github.com/apache/airflow/pull/40017>. > > > > Discuss: Workstreams & workstream owners (Airflow 3 Workstreams > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams > >) > > Updates on the *existing AIPs for the 3.0* list > > - AIP-66 (DAG Versioning—Execution side): Jedidiah Cunningham will have > the draft by the next dev call. > - AIP-72 (Task Execution SDK) : No updates this week. Ash > Berlin-Taylor aims to have the VOTE open for it by the next dev call. > - AIP-38 (UI Modernization): Brent is focussing on Airflow 2.10 > deliverables; post that, he will focus on replacing this AIP with a new > Umbrella AIP. There was a consensus on the call that we should split > this > AIP into multiple AIPs (with an Umbrella AIP) as below: > > - React'ifying the UI & changing the UX required for a modern Webserver > - FAB removal: The removal of Flask Appbuilder will have backend > implications such as changes needed to Authentication (currently, it > is > done via FAB Auth backends > <https://flask-appbuilder.readthedocs.io/en/latest/security.html>.), > changes to the Plugin interface, custom API endpoints etc. > - New Features like DAG Folders/DAG Groups > > - AIP-57 Refactor SLA Feature: The initial AIP was created with 2.x > in mind, so it should be revised. Some things might be easier with 3.0, > as > we can make breaking changes. One of the contention points last time was > around whether the SLA duration time should start from DAG/task start or > should be based on absolute time, as reflected in this PR comment > <https://github.com/apache/airflow/pull/36639#issuecomment-2021080626>. > Shubham > Mehta and Kaxil Naik will reach out to Sungwon Yun to see if he is > interested in being part of this effort. > - AIP-65 (DAG Versioning—UI side): Before the next dev call, Jedidiah > Cunningham will decide whether to keep this AIP or make it part of any > existing DAG Versioning or UI Modernization AIP. > - AIP-67 (Multi-team): Jarek Potiuk will revise this AIP based on all > the discussions and new AIPs to present the overview in the next dev > call. Shubham Mehta and his team are planning to co-own/contribute to > this > AIP with Jarek. > - AIP-68 (Extending plugin interface): Jens Scheffler & Brent will > create a POC PR to determine the direction to take. Jens will do it > before > the next dev call to then get the final feedback on 2.10.x vs 3.0. > - AIP-69 (Remote Executor): Jens Scheffler organized a call with > interested contributors a day after the dev call to get feedback. A > summary > of it is posted here > <https://lists.apache.org/thread/h2nxkto0lxgjnqj8yps0qsh7ppbccx6g>. > > > > Updates on the "*other candidates for 3.0*" list: > > - *Enhanced Data Awareness*: Tzu-ping Chung and Constance > Martineau created draft AIPs (AIP-74 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-74+Introducing+Data+Assets > > > , AIP-75 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-75+New+Asset-Centric+Syntax > > > , AIP-76 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-76+Asset+Partitions > > > , AIP-77 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-77+Asset+Validations > >) > for this workstream (under Umbrella AIP: AIP-73 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-73+Expanded+Data+Awareness > >). > These AIPs are marked for Airflow 3.0 but AIP-77 might move to Airflow > 3.1, > TBD for now. > - *Scheduler-Managed Backills *(aka Backfills at scale): Daniel > Standish has an internal draft for it that will be published around next > week. > - *Poll external Datasets to have event-based DAG scheduling: *Vincent > BECK volunteered to own this epic for *3.0*. He and Shubham Mehta plan > to publish a draft AIP in the next 2 weeks. > - Dennis Ferruzzi has volunteered to own the "*Inspect & Simplify > Airflow Configurations*" & "*Inspect & Revamp the cardinality of > Metrics*" > streams. > - *Synchronous DAG Execution*: No update this week but targeting a > proposal after the next dev call > - *Make Execution Date non-unique for a DAG*: No update for now; TP is > focussing on the Data Awareness AIPs. > - *Scheduler Performance Improvements*: Some areas in the scheduler have > been identified for improvements. If anyone has any specific areas of > the > Scheduler loop that they would like to own, please add them to the > list on Airflow > 3 Workstreams > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams > >. > The performance goals would be tied to the specific area of optimization > rather than the scheduler as a whole. > - *Consider Developing Airflow and Providers separately*. In the coming > weeks, Kaxil Naik will draft a pro-cons list in a document to send to > the > mailing list around this epic. > - *Respect permissions in CLI*: Buğra Öztürk has volunteered to own this > stream and, ideally, will start working on a proposal next. > - *Improve security of Airflow Supply Chain*: Once things are more > defined, Jarek Potiuk will send a document about this stream of work > describing its impact on dependencies in the coming month. > - *Observability of Callbacks on UI*: Some of this might be covered in > AIP-72 & AIP-69 but we will keep this item in the table so we don't > forget > about it. > - *Remove StatsD and replace it with Prometheus as a first-class > citizen*: > We agreed that we should replace StatsD with OpenTelemetry and not > Prometheus. It needs an owner who could figure out the impact of > breaking > changes since StatsD metrics are utilized by all users, including > Airflow > service providers, to monitor and alert on the health of Airflow > deployments. > - *Overhaul Operator Templating behaviour*: Tzu-ping Chung mentioned he > would be happy to write the design document & asked for someone to work > on > the implementation. Shahar Epstein expressed his interest in working on > this effort. > - Shubham Mehta expressed interest in working on a native DAG Factory > that is currently on the 3.1 list. We would keep this for 3.1 for now > and > reevaluate based on the progress of other epics. > - There have been no noteworthy updates on any other streams. Kaxil > Naik will create GitHub issues for items in that table with no owners > and > post them on the mailing list to see if anyone is interested in leading > it. > If there are no takers by the end of August, they will be moved to the > 3.1+ > list. > > > The next dev call will be on 11 Jul 2024. >
