This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 98aa84b6c0 Limit aibotocore to help `pip` with backtracking issue for
s3fs (#35952)
98aa84b6c0 is described below
commit 98aa84b6c038215455897ff6cd7abba9e5ab27cc
Author: Jarek Potiuk <[email protected]>
AuthorDate: Wed Nov 29 14:12:16 2023 +0100
Limit aibotocore to help `pip` with backtracking issue for s3fs (#35952)
Recent release of aiobotocore, boto3, botocore conflicts with
s3fs and confuses `pip` enough to go into conflict or long
backtracking.
---
Dockerfile.ci | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index a22c363fa5..b9f4584da4 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -1225,7 +1225,23 @@ RUN echo "Airflow version: ${AIRFLOW_VERSION}"
# Without grpcio-status limit, pip gets into very long backtracking
# We should attempt to remove it in the future
#
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="grpcio-status>=1.55.0
aiobotocore>=2.7.0"
+# Aiobotocore is limited for eager upgrade because it either causes a long
backtracking or
+# conflict when we do not limit it. It seems that `pip` has a hard time
figuring the right
+# combination of dependencies for aiobotocore, botocore, boto3 and s3fs
together
+#
+# The root cause of the problem is that botocore (1.33.2), boto3 (1.33.2),
aibotocore (2.8.0) all
+# released on November 28/29 2023. They are all compatible with each other but
they
+# do not have corresponding s3fs release that would be compatible with them.
The
+# s3fs latest 2023.10.0 release has iobotocore ~=2.7.0 which effectively
forbids it to use 2.8.0
+# (and friends) and it confuses `pip` enough that it's not able to figure out
that latest s3fs has a working
+# combination of aibotocore,botocore,boto3,s3fs - and it either gets into
conflict or backtracks for hours.
+#
+# We need to help `pip` a little bit by limiting
+# limiting aiobotocore (limiting s3fs to latest version causes conflict)
+# This limit should be removed once new version of s3fs is released that is
compatible with
+# aiobotocore 2.8.0 (beyond 2023.10.0).
+#
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="grpcio-status>=1.55.0
aiobotocore>=2.7.0,<2.8.0"
ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
ARG VERSION_SUFFIX_FOR_PYPI=""