Hi all, I have created a document to summarize the discussion from our second dev call for Airflow 2.0.
Thank you all who joined the call. *Doc Link*: https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree to anything in the Summary please voice your opinion. Including the Summary here too (might potentially break formatting): *Key Decisions* - *Smart Sensors – *in 2.0 or 2.1 - AIP-17 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor> | PR: https://github.com/apache/airflow/pull/5499 - We have not come to a conclusion yet on whether this should be included in 2.0 or not. The majority is towards adding it in 2.0 (as it supports Airflow 2.0's Scalability story) and marking it as *experimental*. - There were some questions raised around supporting this new feature. So we decided that *everyone would take a look at the PR itself and we will spend a few minutes in the next meeting to decide whether it is 2.0 or not*. - *Simplification of KubernetesExecutor / KubernetesPodOperator* - PR: https://github.com/apache/airflow/pull/10393 - This will be part of *Airflow 2.0* - *Airflow Upgrade Check* (airflow upgrade-check)* command * - WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design Doc: https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv - *Scope*: - Users bash script won’t be included but anything in the core Airflow would be covered - *DAG Definitions*: - Changes in Path for contrib to Providers packages - DAG Interfaces: changes in arguments of a DAG / BaseOperator - *Configurations*: - Option to auto-replace deprecated configs with new options - *Run-time Core items*: - Changes like "Connection type can't be null". The upgrade-check should at least shown warning if it can't provide option to detect the type. - *CLI refactor is out-of-scope* - Automatic refactor is *out-of-scope* as it is too difficult to cover all the cases in the Users bash scripts. - This will be covered by docs or by showing warnings via the upgrade-check command - *Experimental API to New API refactor is out-of-scope* (will be covered by Migration docs) - We agreed that the airflow upgrade-check command *needs to be available in the last release before Airflow 2.0* (1.10.x or 1.11.x) - Potential problems with time-consuming DB Migration were also discussed. If we identify such a DB Migration (example the one involving TaskInstance table) should be noted separately in Updating.md to provide a warning to the users. - *DEV Calls Feedback* - We agreed on having *Weekly calls from 7 September onwards* - Calls will start with a 5-min reviewing the progress from the last call towards 2.0 - *Process* - A *2.0.0-test* branch will be created on 10 Sep 2020 - Changelog: - The current way of Changelog is OK. We don't need further categorization like Webserver, Scheduler etc. - Separate Changelog would be created for Providers Packages - We need to figure a way to tag/label PRs & Issues with correct categories. Some options that were discussed were: - Adding labels on the PRs & Issues via Bot - A field in PR template for PR authors to add, the bot would then read the field which would be used to label the PR - Add rules, for example Committers needs to add appropriate labels to the PR before merging it. We could have a scheduled Github Actions workflow that would fail if it finds PRs without labels. *Things to Discuss Next* - *7 September* - Progress, Current Work & Discussions - API - Providers Packages - Discuss open questions - Improvements to SubDags / Concept of TaskGroup - AIP-34 <https://github.com/apache/airflow/pull/10153> - *14 September* - Process: - When should we defer the in-scope items to post-2.0 - Completion by a date? - Progress by a date? - Progress, Current Work & Discussions - Scheduler HA - Docs Improvements - Helm Chart - Discuss the issue with sources Regards, Kaxil
