potiuk commented on code in PR #25306:
URL: https://github.com/apache/airflow/pull/25306#discussion_r930933847
##########
dev/provider_packages/prepare_provider_packages.py:
##########
@@ -314,7 +314,20 @@ def get_install_requirements(provider_package_id: str,
version_suffix: str) -> s
:return: install requirements of the package
"""
- install_requires = ALL_DEPENDENCIES[provider_package_id][DEPS]
+
+ def apply_version_suffix(install_clause: str) -> str:
+ if install_clause.startswith("apache-airflow") and ">=" in
install_clause and version_suffix != "":
+ # This is workaround for `pip` bug. When you specify dependency as
>= X.Y.Z, and you
+ # Have packages X.Y.Zdev0 or X.Y.Zrc1, the "=" part does not
include the pre-release versions
+ # you need to explicitly specify >= X.Y.Za0 to include pre-release
versions.
+ # Therefore, when we are preparing pre-releases we always add a0
for airflow and other providers
+ # to Allow installing two packages released in rc* that depend on
each-other
Review Comment:
I tried to use pre (I will have a nice feature added to docker files out of
that), but I am afraid this is indeed the bug and I remembered our earlier
discussion well. This is what happened when I tried to install the providers:
```
pip install --find-links=file:///docker-context-files
--root-user-action ignore --upgrade --upgrade-strategy eager
--pre
'apache-airflow[amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]==2.4.0.dev0'
/docker-context-files/apache_airflow_providers_amazon-4.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_celery-3.0.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_cncf_kubernetes-4.2.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_common_sql-1.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_docker-3.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_elasticsearch-4.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_ftp-3.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_google-8.2.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_grpc-3.0.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_hashicorp-3.0.1.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_http-4.0.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_imap-3.0.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_microsoft_azure-4.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_mysql-3.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_odbc-3.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_postgres-5.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_redis-3.0.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_sendgrid-3.0.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_sftp-4.0.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_slack-5.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_sqlite-3.1.0.dev0-py3-none-any.whl
/docker-context-files/apache_airflow_providers_ssh-3.1.0.dev0-py3-none-any.whl
'dill<0.3.3'
```
This fails with:
```
#56 34.54 ERROR: Cannot install apache-airflow-providers-amazon==4.1.0.dev0,
apache-airflow-providers-common-sql 1.1.0.dev0 (from
/docker-context-files/apache_airflow_providers_common_sql-1.1.0.dev0-py3-none-any.whl),
apache-airflow-providers-elasticsearch==4.1.0.dev0,
apache-airflow-providers-google==8.2.0.dev0 and
apache-airflow[amazon,async,celery,cncf-kubernetes,dask,docker,elasticsearch,ftp,google,google-auth,grpc,hashicorp,http,ldap,microsoft-azure,mysql,odbc,pandas,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]==2.4.0.dev0
because these package versions have conflicting dependencies.
#56 34.54
#56 34.54 The conflict is caused by:
#56 34.54 The user requested apache-airflow-providers-common-sql
1.1.0.dev0 (from
/docker-context-files/apache_airflow_providers_common_sql-1.1.0.dev0-py3-none-any.whl)
#56 34.54
apache-airflow[amazon,async,celery,cncf-kubernetes,dask,docker,elasticsearch,ftp,google,google-auth,grpc,hashicorp,http,ldap,microsoft-azure,mysql,odbc,pandas,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]
2.4.0.dev0 depends on apache-airflow-providers-common-sql
#56 34.54 apache-airflow-providers-amazon 4.1.0.dev0 depends on
apache-airflow-providers-common-sql
#56 34.54 apache-airflow-providers-elasticsearch 4.1.0.dev0 depends on
apache-airflow-providers-common-sql
#56 34.54 apache-airflow-providers-google 8.2.0.dev0 depends on
apache-airflow-providers-common-sql>=1.1.0
#56 34.54
#56 34.54 To fix this you could try to:
#56 34.54 1. loosen the range of package versions you've specified
#56 34.54 2. remove package versions to allow pip attempt to solve the
dependency conflict
#56 34.54
#56 34.54 ERROR: ResolutionImpossible: for help visit
https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
```
The problem is that
* apache-airflow-providers-google 8.2.0.dev0 depends on
apache-airflow-providers-common-sql>=1.1.0
* The user requested apache-airflow-providers-common-sql 1.1.0.dev0 (from
/docker-context-files/apache_airflow_providers_common_sql-1.1.0.dev0-py3-none-any.whl
I was using ``--pre`` flag (as you can see above).
Still google's > 1.1.0 conflicts with 1.1.0dev0 - which is the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
