potiuk commented on issue #15463: URL: https://github.com/apache/airflow/issues/15463#issuecomment-824300116
Just to explain the root cause of those problems and some complexities involved 1) Root cause of the problem above: The original problem turned out to be a bug in recently changed mechanism of generating constraints. Fixed here: https://github.com/apache/airflow/pull/15470 - the bug was, that when we prepared constraints for "production" release we use "pip-providers" version of constraints - we assume that we install all providers that are **currently** (at the time of constraint generation) available in PyPI and work out the version of constraints that will allow to install all of them. The problem was that when we calculated those, we did not include few extras that have no providers. So rather than calculating the set of constraints that are "good" for `pip install apache-airflow[google_auth,ldap....] apache-airflow-providers-*` we calculated what is "good" for `pip install apache-airflow-providers-*`. This is why oauhlib was bumped (wrongly). 2) Now there are other problems that are more subtle, the problem is on how PIP treats "extra" dependencies especially in transitive dependencies. Those are differently solved at "installation" time and differently "at rest" and differently when different PIP versions are used. This gets complex fast. Example (real that we have now) : `airflow[google] -> apache-airflow-providers-google -> pandas-gbq-> "google-cloud-bigquery[bqstorage,pandas]>=1.11.1,<3.0.0dev,!=2.4.*"` and then it gets funny... because what we have in `google-cloud-bigguery` in extras is: ``` `"bqstorage": [ "google-cloud-bigquery-storage >= 2.0.0, <3.0.0dev", ``` There are problems with that * old pip does not complain about the "bqstorage" limitation - and will happily install goog-le-coloud -bigiquery-storage outside of the range * new pip > 20.3 does complain and will not install * however when they are installed both pip's after installation when you run 'pip check` with non-matching google-cloud-biguery will show that everything is fine - because pandas-gbq -> google-cloud-bigquery[bqstorage,pandas] is transitive, only valid at installation time. It's not 'install-requires' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
