potiuk commented on PR #41626:
URL: https://github.com/apache/airflow/pull/41626#issuecomment-2301454608
> is this for opensearch? or opensearch is part of apache-beam
This is because currently the tests for "latest" botocore have conflicting
dependencies and we are workarounding it - by temporarily disabling
dependencies that have those conflicts.
In this case the problem is `apache.beam` that (currently released version)
has reqeusts < 3 but != 2.32.* (due to some problem with their test harness -
apparently requests did not work with the docker test harness they used). The
problem with that is that growing number of our dependencies (for example
opensearch) has `requests >= 2.32` and when we attempt to upgrade botocore -
through a ripple-effect of dependencies pulling other dependencies, this leads
to unsolvable conflicts (because latest botocore will not work with earlier
versions of those dependencies that can work with requests 2.31.*:
```
Uninstalled 3 packages in 320ms
- aiobotocore==2.13.2
- s3fs==2024.6.1
- yandexcloud==0.313.0
+ uv pip install --python /usr/local/bin/python --upgrade boto3 botocore
'oss2>=2.14.0' 'cryptography<43.0.0' 'requests!=2.32.*,<3.0.0,>=2.24.0'
Resolved 19 packages in 32.61s
Prepared 3 packages in 325ms
Uninstalled 3 packages in 913ms
Installed 3 packages in 140ms
- boto3==1.34.131
+ boto3==1.35.2
- botocore==1.34.131
+ botocore==1.35.2
- requests==2.32.3
+ requests==2.31.0
+ set +x
opensearch-py 2.7.0 has requirement requests<3.0.0,>=2.32.0, but you have
requests 2.31.0.
```
The apache-beam problem has been solved and merged as I opened
https://github.com/apache/beam/issues/32080 - 2 weeks ago so that they remove
the limitation - and they merged it literally yesterday -
https://github.com/apache/beam/pull/32236
So by removing opensearch from latest botocore tests, we termporarily make
it works now - but once apache-beam 2.59.0 is released, we should be able to
bring opensearch back to those tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
