uranusjr opened a new issue #18705: URL: https://github.com/apache/airflow/issues/18705
I think this is the problem hit by #18684. I am not entirely familiar with the CI script, but this is my best guess reading the CI logs. https://github.com/apache/airflow/pull/18684/checks?check_run_id=3788521722 cc @eladkal @potiuk ## The problem Currently the *Installing all packages with eager upgrade* job installs `airflow[devel_ci]` with `--upgrade-strategy=eager`, which installs the latest versions of all Airflow providers. However, those providers are specified by name, so the "latest version" is defined as the version on PyPI, with whatever the dependencies defined in those released versions. This is usually not a problem, but if we change any of the providers’ dependencies to a new range mutually exclusive with the released provider version, this eager-upgrade job would fail because it’s impossible to both satisfy the new version range included as a part of the `devel_ci` extra, and the old version range included by the provider release from PyPI. This becomes an issue in e.g. #18684, where a version range (`sshtunnel>=0.1.4,<0.2`) is changed to an entirely non-overlapping one (`sshtunnel>=0.3.2,<0.5`). ## Proposed solution Before running the *eager upgrade* job, CI should first build all local providers into installable distributions (perferrably wheels), and constrain the `pip install` command like this: ``` # provider-constraints.txt apache-airflow-providers-airbyte @ ./provider-wheels/apache_airflow_providers_airbyte-...whl # ... all the wheel paths ``` ```shell pip install \ --upgrade \ --upgrade-strategy eager \ '.[devel_ci]' \ 'lazy-object-proxy<1.5.0' \ 'pyjwt<2.0.0' \ 'dill<0.3.3' \ 'certifi<2021.0.0' \ -c provider-constraints.txt # This line is new. ``` So providers are always fetched locally instead of from PyPI, thus always include the most up-to-date dependencies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
