Yeah that sounds good too. On Thu, Feb 11, 2021, 15:17 Ash Berlin-Taylor <[email protected]> wrote:
> The problem with tightly specifying all the constraints (including the > providers) is that it means you can't do something like `pip install -U > apache-airflow-providers-google` but have the _core_ Airflow constrained. > (Pip may be better at upgrading less in cases like this now?) > > I have a proposal: two constraint files (for each python version) -- a > "core" and a "full". > > The "full" is as you propose, with the providers, and their deps in the > file. > > The "core" is _just_ the core requirements for Airflow without any > providers, or any transitive deps. This will include deps non-provider > extras though. > > How does that sound? > > -ash > > On Thu, 11 Feb, 2021 at 14:07, Kaxil Naik <[email protected]> wrote: > > Yup, that is correct. That will allow us to make sure that whenever > Airflow was released, all the dependencies including the provider are > snapshotted in constraints. So even if someone tries to install the same > version a year later with constraints it should work fine without having to > worry about the latest version of a specific provider breaking it. > > And then users can ofcourse install or upgrade providers after that if > they like. > > Kaxil - did I understand it correctly ? If so - I think this is the best >> we can do to keep two properties: > > * repeatable installation of already released version >> * capability (and easy way of) upgrading to latest providers > > > On Thu, Feb 11, 2021 at 12:42 PM Jarek Potiuk <[email protected]> wrote: > >> > Oh I misunderstood -- I thought you were suggesting putting the >> transitive deps of apache-airflow-providers-google v2.0 into constraint-2.0 >> files etc. >> >> Well. That too. The transitive deps already are in the constraint files >> and that will remain, I think this is the main reason why we have >> the constraint files. The main reason why constraint files are "snapshots >> of all dependencies'' (currently they exclude providers) is to have a >> repeatable installation. Let me reiterate then how I understand Kaxil's >> proposal (which I think makes perfect sense). >> >> I really see the "extras" and constraints as a convenient way for users >> to install a released version of airflow with the set of providers they >> choose and dependencies in versions that we know are working. No more, no >> less. Then they are free to upgrade the dependencies as they wish. >> >> How I see the current proposal - the constraint files will only differ >> from the current ones by adding 'apache-airflow-providers-google==1.0.0" >> for example. Literally (compared to the current process) it means that we >> will just add the version of providers that were released at the time the >> airflow X.Y.Z version was released (this is one line change in generation >> of constraints as we have now). So the final constraint file for 2.0.1 >> version will look like this: >> >> apache-airfow-providers-google=1.0.0 >> google-cloud-automl=1.9.0 >> .... >> ~500 other dependencies with == >> ... >> >> Those constraint files will contain all providers that were released at >> the time of airflow X.Y.Z release and all their transitive dependencies. >> This way if you run 'pip apache-airflow[google, amazon]==2.0.1 >> --constraints ...../2.0.1/python3.6.txt ' - you will always get the >> google-provider==1.0.0 installed and amazon 1.0.0 as well. >> >> And that preserves the only capability that constraint files + extras >> give - an easy installation path when you want to install an older version >> of airflow for the first time - with pretty much guarantee that it will >> always work (this is the only problem constraint files were introduced. >> This will be now extended to this semantic: "install airflow x.y.z with all >> the providers and dependencies that we found were ok at the time when x.y.z >> were released". >> >> Then, the users will still be free to do `pip install --upgrade >> apache-airflow-providers-google' and specific upgrade airflow provider to >> the latest version. Or if they are adventurous they could upgrade all >> dependencies to latest with 'pip install apache-airflow[google] --upgrade >> --upgrade-strategy eager' (but without guarantee it will work). >> >> Or if there is a new airflow released they could run: 'pip >> apache-airflow[google, amazon]==2.0.2 --constraints >> ...../2.0.2/python3.6.txt` - and they will get set of dependencies and >> providers that were there at the time of 2.0.2 release (but still they are >> free to upgrade to latest versions of providers at will). >> >> Kaxil - did I understand it correctly ? If so - I think this is the best >> we can do to keep two properties: >> >> * repeatable installation of already released version >> * capability (and easy way of) upgrading to latest providers >> >> >> J. >> >> >> >> On Thu, Feb 11, 2021 at 1:06 PM Ash Berlin-Taylor <[email protected]> wrote: >> >>> Oh I misunderstood -- I thought you were suggesting putting the >>> transitive deps of apache-airflow-providers-google v2.0 in to >>> constraint-2.0 files etc. >>> >>> Cool >>> >>> On Thu, 11 Feb, 2021 at 12:11, Jarek Potiuk <[email protected]> wrote: >>> >>> > This unfortunately means that people would be unable to install v1 of >>> the google provider anymore -- forcing them to upgrade. >>> >>> Not really. If we specify just this: [google] -> >>> "apache-airflow-providers-google" - any provider version could be installed. >>> >>> >>> On Thu, Feb 11, 2021 at 12:07 PM Ash Berlin-Taylor <[email protected]> >>> wrote: >>> >>>> On Thu, 11 Feb, 2021 at 00:34, Jarek Potiuk <[email protected]> wrote: >>>> >>>> *Solution proposal:* >>>> >>>> Every time when we release a new wave of providers, we regenerate the >>>> constraints for all past released 2.* versions of airflow, so that the new >>>> providers are taken into account and they can install cleanly with `pip >>>> install apache-airflow[provider]==2.0.N --constraint == .... 2.0.N/python >>>> ... >>>> >>>> Both problems can be solved rather easily. 1) requires 2.0.2 release of >>>> Airflow, 2) can be implemented any time (happy to do it). >>>> >>>> Let me know what you think. >>>> >>>> >>>> This unfortunately means that people would be unable to install v1 of >>>> the google provider anymore -- forcing them to upgrade. >>>> >>>> I'm not sure there's a _ready_ solution to this though. >>>> >>>> -ash >>>> >>> >>> >>> -- >>> +48 660 796 129 >>> >>> >> >> -- >> +48 660 796 129 >> >
