And we have a new addition from Kamil about the need to extend slightly plugin mechanism to be able to cover dynamically "Connections", "Connection Form" and "Extra Links" - those are indeed the "core -> Providers" dependencies that we still have.
They seem to be easy to handle by making providers "plugins" and extending the plugin mechanism a bit. Thanks Kamil for the thoughtful input! On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <[email protected]> wrote: > Just a short reminder - for some more comments/review on the "PIP package > model of Airflow 2.0" doc > > https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit# > > I've added one small addition - in this model we want to make sure that > there are no dependencies of core packages on any of the providers - we do > not run such checks yet but it's easy to add. > > J > > > On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk <[email protected]> > wrote: > >> Cool! >> >> If you have comments on particular sections/paragraphs - it's easier to >> keep track of it and respond in the doc. If you have some general >> statements, and some summary of your thinking after the review - it's best >> to respond to the email :) >> >> I am ok with both and will aggregate it eventually. >> >> J. >> >> >> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <[email protected]> wrote: >> >>> Jarek, >>> >>> Thank you, this is very helpful. >>> I assume that you would like comments in the document itself? >>> Or, would you like them in email? >>> >>> Best regards, >>> Vikram >>> >>> >>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <[email protected]> >>> wrote: >>> >>> > As promised during the last call I prepared the proposal on how we can >>> > approach the package model for Airflow 2.0 - including the "Provider >>> > Packages" approach. >>> > >>> > https://s.apache.org/airflow-2-0-package-model >>> > >>> > I would like to discuss it at our next meeting on Monday. I'd love to >>> > hear your comments. >>> > >>> > J. >>> > >>> > >>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk < >>> [email protected]> >>> > wrote: >>> > > >>> > > +1 Kevin on the call :). >>> > > >>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <[email protected]> >>> wrote: >>> > >> >>> > >> Thanks Kevin, Looking forward to see you on the next call. >>> > >> >>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <[email protected]> wrote: >>> > >> >>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at >>> > least the >>> > >> > next meeting trying to see if we can provide more perspectives on >>> > >> > SmartSensor and anything else we can help. >>> > >> > >>> > >> > >>> > >> > Cheers, >>> > >> > Kevin Y >>> > >> > >>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <[email protected]> >>> > wrote: >>> > >> > >>> > >> > > Hi all, >>> > >> > > >>> > >> > > I have created a document to summarize the discussion from our >>> > second dev >>> > >> > > call for Airflow 2.0. >>> > >> > > >>> > >> > > Thank you all who joined the call. >>> > >> > > >>> > >> > > *Doc Link*: >>> > >> > > >>> > >> > > >>> > >> > >>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020 >>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020> >>> > < >>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >>> > >>> > >> > < >>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >>> > > >>> > >> > > < >>> > >> > >>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >>> > >> > > >>> > >> > > >>> > >> > > To all those who attended, can you please double-check and add >>> if I >>> > have >>> > >> > > missed anything? >>> > >> > > >>> > >> > > To all those who didn't join, if you disagree to anything in the >>> > Summary >>> > >> > > please voice your opinion. >>> > >> > > >>> > >> > > Including the Summary here too (might potentially break >>> formatting): >>> > >> > > >>> > >> > > *Key Decisions* >>> > >> > > >>> > >> > > - *Smart Sensors – *in 2.0 or 2.1 >>> > >> > > - AIP-17 >>> > >> > > < >>> > >> > > >>> > >> > >>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor >>> > >> > > > >>> > >> > > | >>> > >> > > PR: https://github.com/apache/airflow/pull/5499 >>> > >> > > - We have not come to a conclusion yet on whether this >>> should >>> > be >>> > >> > > included in 2.0 or not. The majority is towards adding it >>> in >>> > 2.0 >>> > >> > (as >>> > >> > > it >>> > >> > > supports Airflow 2.0's Scalability story) and marking it >>> as >>> > >> > > *experimental*. >>> > >> > > - There were some questions raised around supporting this >>> new >>> > >> > > feature. So we decided that *everyone would take a look at >>> > the PR >>> > >> > > itself and we will spend a few minutes in the next >>> meeting to >>> > >> > decide >>> > >> > > whether it is 2.0 or not*. >>> > >> > > - *Simplification of KubernetesExecutor / >>> KubernetesPodOperator* >>> > >> > > - PR: https://github.com/apache/airflow/pull/10393 >>> > >> > > - This will be part of *Airflow 2.0* >>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command * >>> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 >>> | >>> > Design >>> > >> > > Doc: >>> > >> > > >>> > >> > > >>> > >> > >>> > >>> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv >>> > >> > > - *Scope*: >>> > >> > > - Users bash script won’t be included but anything in >>> the >>> > core >>> > >> > > Airflow would be covered >>> > >> > > - >>> > >> > > >>> > >> > > *DAG Definitions*: >>> > >> > > - Changes in Path for contrib to Providers packages >>> > >> > > - DAG Interfaces: changes in arguments of a DAG / >>> > >> > BaseOperator >>> > >> > > - *Configurations*: >>> > >> > > - Option to auto-replace deprecated configs with new >>> > options >>> > >> > > - *Run-time Core items*: >>> > >> > > - Changes like "Connection type can't be null". The >>> > >> > > upgrade-check should at least shown warning if it >>> can't >>> > >> > > provide option to >>> > >> > > detect the type. >>> > >> > > - *CLI refactor is out-of-scope* >>> > >> > > - Automatic refactor is *out-of-scope* as it is too >>> > difficult >>> > >> > > to cover all the cases in the Users bash scripts. >>> > >> > > - This will be covered by docs or by showing >>> warnings >>> > via the >>> > >> > > upgrade-check command >>> > >> > > - *Experimental API to New API refactor is >>> out-of-scope* >>> > (will >>> > >> > be >>> > >> > > covered by Migration docs) >>> > >> > > - We agreed that the airflow upgrade-check command *needs >>> to >>> > be >>> > >> > > available in the last release before Airflow 2.0* (1.10.x >>> or >>> > >> > 1.11.x) >>> > >> > > - Potential problems with time-consuming DB Migration were >>> also >>> > >> > > discussed. If we identify such a DB Migration (example the >>> one >>> > >> > involving >>> > >> > > TaskInstance table) should be noted separately in >>> Updating.md to >>> > >> > > provide a >>> > >> > > warning to the users. >>> > >> > > - *DEV Calls Feedback* >>> > >> > > - We agreed on having *Weekly calls from 7 September >>> onwards* >>> > >> > > - Calls will start with a 5-min reviewing the progress >>> from >>> > the >>> > >> > last >>> > >> > > call towards 2.0 >>> > >> > > - *Process* >>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020 >>> > >> > > - Changelog: >>> > >> > > - The current way of Changelog is OK. We don't need >>> further >>> > >> > > categorization like Webserver, Scheduler etc. >>> > >> > > - Separate Changelog would be created for Providers >>> > Packages >>> > >> > > - We need to figure a way to tag/label PRs & Issues >>> with >>> > correct >>> > >> > > categories. Some options that were discussed were: >>> > >> > > - Adding labels on the PRs & Issues via Bot >>> > >> > > - A field in PR template for PR authors to add, the >>> bot >>> > would >>> > >> > > then read the field which would be used to label >>> the PR >>> > >> > > - Add rules, for example Committers needs to add >>> > appropriate >>> > >> > > labels to the PR before merging it. We could have a >>> > >> > > scheduled Github >>> > >> > > Actions workflow that would fail if it finds PRs >>> without >>> > >> > > labels. >>> > >> > > >>> > >> > > *Things to Discuss Next* >>> > >> > > >>> > >> > > - *7 September* >>> > >> > > - Progress, Current Work & Discussions >>> > >> > > - API >>> > >> > > - Providers Packages >>> > >> > > - Discuss open questions >>> > >> > > - Improvements to SubDags / Concept of TaskGroup >>> > >> > > - AIP-34 < >>> https://github.com/apache/airflow/pull/10153> >>> > >> > > - *14 September* >>> > >> > > - Process: >>> > >> > > - When should we defer the in-scope items to post-2.0 >>> > >> > > - Completion by a date? >>> > >> > > - Progress by a date? >>> > >> > > - Progress, Current Work & Discussions >>> > >> > > - Scheduler HA >>> > >> > > - Docs Improvements >>> > >> > > - Helm Chart >>> > >> > > - Discuss the issue with sources >>> > >> > > >>> > >> > > >>> > >> > > Regards, >>> > >> > > Kaxil >>> > >> > > >>> > >> > >>> > > >>> > > >>> > > >>> > > -- >>> > > >>> > > Jarek Potiuk >>> > > Polidea | Principal Software Engineer >>> > > >>> > > M: +48 660 796 129 >>> > > >>> > > >>> > >>> > >>> > -- >>> > >>> > Jarek Potiuk >>> > Polidea | Principal Software Engineer >>> > >>> > M: +48 660 796 129 >>> > >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
