Yup, that is correct. That will allow us to make sure that whenever Airflow was released, all the dependencies including the provider are snapshotted in constraints. So even if someone tries to install the same version a year later with constraints it should work fine without having to worry about the latest version of a specific provider breaking it.
And then users can ofcourse install or upgrade providers after that if they like. Kaxil - did I understand it correctly ? If so - I think this is the best we > can do to keep two properties: * repeatable installation of already released version > * capability (and easy way of) upgrading to latest providers On Thu, Feb 11, 2021 at 12:42 PM Jarek Potiuk <[email protected]> wrote: > > Oh I misunderstood -- I thought you were suggesting putting the > transitive deps of apache-airflow-providers-google v2.0 into constraint-2.0 > files etc. > > Well. That too. The transitive deps already are in the constraint files > and that will remain, I think this is the main reason why we have > the constraint files. The main reason why constraint files are "snapshots > of all dependencies'' (currently they exclude providers) is to have a > repeatable installation. Let me reiterate then how I understand Kaxil's > proposal (which I think makes perfect sense). > > I really see the "extras" and constraints as a convenient way for users to > install a released version of airflow with the set of providers they choose > and dependencies in versions that we know are working. No more, no less. > Then they are free to upgrade the dependencies as they wish. > > How I see the current proposal - the constraint files will only differ > from the current ones by adding 'apache-airflow-providers-google==1.0.0" > for example. Literally (compared to the current process) it means that we > will just add the version of providers that were released at the time the > airflow X.Y.Z version was released (this is one line change in generation > of constraints as we have now). So the final constraint file for 2.0.1 > version will look like this: > > apache-airfow-providers-google=1.0.0 > google-cloud-automl=1.9.0 > .... > ~500 other dependencies with == > ... > > Those constraint files will contain all providers that were released at > the time of airflow X.Y.Z release and all their transitive dependencies. > This way if you run 'pip apache-airflow[google, amazon]==2.0.1 > --constraints ...../2.0.1/python3.6.txt ' - you will always get the > google-provider==1.0.0 installed and amazon 1.0.0 as well. > > And that preserves the only capability that constraint files + extras give > - an easy installation path when you want to install an older version of > airflow for the first time - with pretty much guarantee that it will always > work (this is the only problem constraint files were introduced. This will > be now extended to this semantic: "install airflow x.y.z with all the > providers and dependencies that we found were ok at the time when x.y.z > were released". > > Then, the users will still be free to do `pip install --upgrade > apache-airflow-providers-google' and specific upgrade airflow provider to > the latest version. Or if they are adventurous they could upgrade all > dependencies to latest with 'pip install apache-airflow[google] --upgrade > --upgrade-strategy eager' (but without guarantee it will work). > > Or if there is a new airflow released they could run: 'pip > apache-airflow[google, amazon]==2.0.2 --constraints > ...../2.0.2/python3.6.txt` - and they will get set of dependencies and > providers that were there at the time of 2.0.2 release (but still they are > free to upgrade to latest versions of providers at will). > > Kaxil - did I understand it correctly ? If so - I think this is the best > we can do to keep two properties: > > * repeatable installation of already released version > * capability (and easy way of) upgrading to latest providers > > > J. > > > > On Thu, Feb 11, 2021 at 1:06 PM Ash Berlin-Taylor <[email protected]> wrote: > >> Oh I misunderstood -- I thought you were suggesting putting the >> transitive deps of apache-airflow-providers-google v2.0 in to >> constraint-2.0 files etc. >> >> Cool >> >> On Thu, 11 Feb, 2021 at 12:11, Jarek Potiuk <[email protected]> wrote: >> >> > This unfortunately means that people would be unable to install v1 of >> the google provider anymore -- forcing them to upgrade. >> >> Not really. If we specify just this: [google] -> >> "apache-airflow-providers-google" - any provider version could be installed. >> >> >> On Thu, Feb 11, 2021 at 12:07 PM Ash Berlin-Taylor <[email protected]> >> wrote: >> >>> On Thu, 11 Feb, 2021 at 00:34, Jarek Potiuk <[email protected]> wrote: >>> >>> *Solution proposal:* >>> >>> Every time when we release a new wave of providers, we regenerate the >>> constraints for all past released 2.* versions of airflow, so that the new >>> providers are taken into account and they can install cleanly with `pip >>> install apache-airflow[provider]==2.0.N --constraint == .... 2.0.N/python >>> ... >>> >>> Both problems can be solved rather easily. 1) requires 2.0.2 release of >>> Airflow, 2) can be implemented any time (happy to do it). >>> >>> Let me know what you think. >>> >>> >>> This unfortunately means that people would be unable to install v1 of >>> the google provider anymore -- forcing them to upgrade. >>> >>> I'm not sure there's a _ready_ solution to this though. >>> >>> -ash >>> >> >> >> -- >> +48 660 796 129 >> >> > > -- > +48 660 796 129 >
