Just a short reminder - for some more comments/review on the "PIP package model of Airflow 2.0" doc https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
I've added one small addition - in this model we want to make sure that there are no dependencies of core packages on any of the providers - we do not run such checks yet but it's easy to add. J On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk <[email protected]> wrote: > Cool! > > If you have comments on particular sections/paragraphs - it's easier to > keep track of it and respond in the doc. If you have some general > statements, and some summary of your thinking after the review - it's best > to respond to the email :) > > I am ok with both and will aggregate it eventually. > > J. > > > On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <[email protected]> wrote: > >> Jarek, >> >> Thank you, this is very helpful. >> I assume that you would like comments in the document itself? >> Or, would you like them in email? >> >> Best regards, >> Vikram >> >> >> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <[email protected]> >> wrote: >> >> > As promised during the last call I prepared the proposal on how we can >> > approach the package model for Airflow 2.0 - including the "Provider >> > Packages" approach. >> > >> > https://s.apache.org/airflow-2-0-package-model >> > >> > I would like to discuss it at our next meeting on Monday. I'd love to >> > hear your comments. >> > >> > J. >> > >> > >> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <[email protected] >> > >> > wrote: >> > > >> > > +1 Kevin on the call :). >> > > >> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <[email protected]> >> wrote: >> > >> >> > >> Thanks Kevin, Looking forward to see you on the next call. >> > >> >> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <[email protected]> wrote: >> > >> >> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at >> > least the >> > >> > next meeting trying to see if we can provide more perspectives on >> > >> > SmartSensor and anything else we can help. >> > >> > >> > >> > >> > >> > Cheers, >> > >> > Kevin Y >> > >> > >> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <[email protected]> >> > wrote: >> > >> > >> > >> > > Hi all, >> > >> > > >> > >> > > I have created a document to summarize the discussion from our >> > second dev >> > >> > > call for Airflow 2.0. >> > >> > > >> > >> > > Thank you all who joined the call. >> > >> > > >> > >> > > *Doc Link*: >> > >> > > >> > >> > > >> > >> > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020 >> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020> >> > < >> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >> > >> > >> > < >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >> > > >> > >> > > < >> > >> > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >> > >> > > >> > >> > > >> > >> > > To all those who attended, can you please double-check and add >> if I >> > have >> > >> > > missed anything? >> > >> > > >> > >> > > To all those who didn't join, if you disagree to anything in the >> > Summary >> > >> > > please voice your opinion. >> > >> > > >> > >> > > Including the Summary here too (might potentially break >> formatting): >> > >> > > >> > >> > > *Key Decisions* >> > >> > > >> > >> > > - *Smart Sensors – *in 2.0 or 2.1 >> > >> > > - AIP-17 >> > >> > > < >> > >> > > >> > >> > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor >> > >> > > > >> > >> > > | >> > >> > > PR: https://github.com/apache/airflow/pull/5499 >> > >> > > - We have not come to a conclusion yet on whether this >> should >> > be >> > >> > > included in 2.0 or not. The majority is towards adding it >> in >> > 2.0 >> > >> > (as >> > >> > > it >> > >> > > supports Airflow 2.0's Scalability story) and marking it as >> > >> > > *experimental*. >> > >> > > - There were some questions raised around supporting this >> new >> > >> > > feature. So we decided that *everyone would take a look at >> > the PR >> > >> > > itself and we will spend a few minutes in the next meeting >> to >> > >> > decide >> > >> > > whether it is 2.0 or not*. >> > >> > > - *Simplification of KubernetesExecutor / >> KubernetesPodOperator* >> > >> > > - PR: https://github.com/apache/airflow/pull/10393 >> > >> > > - This will be part of *Airflow 2.0* >> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command * >> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 >> | >> > Design >> > >> > > Doc: >> > >> > > >> > >> > > >> > >> > >> > >> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv >> > >> > > - *Scope*: >> > >> > > - Users bash script won’t be included but anything in >> the >> > core >> > >> > > Airflow would be covered >> > >> > > - >> > >> > > >> > >> > > *DAG Definitions*: >> > >> > > - Changes in Path for contrib to Providers packages >> > >> > > - DAG Interfaces: changes in arguments of a DAG / >> > >> > BaseOperator >> > >> > > - *Configurations*: >> > >> > > - Option to auto-replace deprecated configs with new >> > options >> > >> > > - *Run-time Core items*: >> > >> > > - Changes like "Connection type can't be null". The >> > >> > > upgrade-check should at least shown warning if it >> can't >> > >> > > provide option to >> > >> > > detect the type. >> > >> > > - *CLI refactor is out-of-scope* >> > >> > > - Automatic refactor is *out-of-scope* as it is too >> > difficult >> > >> > > to cover all the cases in the Users bash scripts. >> > >> > > - This will be covered by docs or by showing warnings >> > via the >> > >> > > upgrade-check command >> > >> > > - *Experimental API to New API refactor is out-of-scope* >> > (will >> > >> > be >> > >> > > covered by Migration docs) >> > >> > > - We agreed that the airflow upgrade-check command *needs >> to >> > be >> > >> > > available in the last release before Airflow 2.0* (1.10.x >> or >> > >> > 1.11.x) >> > >> > > - Potential problems with time-consuming DB Migration were >> also >> > >> > > discussed. If we identify such a DB Migration (example the one >> > >> > involving >> > >> > > TaskInstance table) should be noted separately in Updating.md >> to >> > >> > > provide a >> > >> > > warning to the users. >> > >> > > - *DEV Calls Feedback* >> > >> > > - We agreed on having *Weekly calls from 7 September >> onwards* >> > >> > > - Calls will start with a 5-min reviewing the progress from >> > the >> > >> > last >> > >> > > call towards 2.0 >> > >> > > - *Process* >> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020 >> > >> > > - Changelog: >> > >> > > - The current way of Changelog is OK. We don't need >> further >> > >> > > categorization like Webserver, Scheduler etc. >> > >> > > - Separate Changelog would be created for Providers >> > Packages >> > >> > > - We need to figure a way to tag/label PRs & Issues with >> > correct >> > >> > > categories. Some options that were discussed were: >> > >> > > - Adding labels on the PRs & Issues via Bot >> > >> > > - A field in PR template for PR authors to add, the >> bot >> > would >> > >> > > then read the field which would be used to label the >> PR >> > >> > > - Add rules, for example Committers needs to add >> > appropriate >> > >> > > labels to the PR before merging it. We could have a >> > >> > > scheduled Github >> > >> > > Actions workflow that would fail if it finds PRs >> without >> > >> > > labels. >> > >> > > >> > >> > > *Things to Discuss Next* >> > >> > > >> > >> > > - *7 September* >> > >> > > - Progress, Current Work & Discussions >> > >> > > - API >> > >> > > - Providers Packages >> > >> > > - Discuss open questions >> > >> > > - Improvements to SubDags / Concept of TaskGroup >> > >> > > - AIP-34 < >> https://github.com/apache/airflow/pull/10153> >> > >> > > - *14 September* >> > >> > > - Process: >> > >> > > - When should we defer the in-scope items to post-2.0 >> > >> > > - Completion by a date? >> > >> > > - Progress by a date? >> > >> > > - Progress, Current Work & Discussions >> > >> > > - Scheduler HA >> > >> > > - Docs Improvements >> > >> > > - Helm Chart >> > >> > > - Discuss the issue with sources >> > >> > > >> > >> > > >> > >> > > Regards, >> > >> > > Kaxil >> > >> > > >> > >> > >> > > >> > > >> > > >> > > -- >> > > >> > > Jarek Potiuk >> > > Polidea | Principal Software Engineer >> > > >> > > M: +48 660 796 129 >> > > >> > > >> > >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea | Principal Software Engineer >> > >> > M: +48 660 796 129 >> > >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
