gameofdatas opened a new issue, #39657: URL: https://github.com/apache/airflow/issues/39657
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.8.1 ### What happened? After upgrading to Airflow 2.8.1, DAGs are failing with a 'No such file or directory' error related to `awscli`. It seems `awscli` needs to be installed as part of the environment setup. However, when adding `awscli` to 'requirements.txt', the installation fails due to version conflicts with dependent packages like `botocore`, `docutils`, `colorama`, and `rsa`. These conflicts arise from the [constraint list](https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt) pinned for Airflow 2.8.1. To match the `botocore` version from the constraint list i.e `1.33.13`, which is a dependent package of awscli, I installed specific version of awscli i.e `1.31.13` installation logs: ``` The conflict is caused by: awscli 1.31.13 depends on docutils<0.17 and >=0.10 The user requested (constraint) botocore==1.33.13 The user requested (constraint) docutils==0.20.1 To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts ``` Although after matching botocore version, docutils and other dependent versions were having version conflict. ### What you think should happen instead? `awscli` should be installed via requirements.txt without any version conflict ### How to reproduce Steps to be followed for reproducing the error: ``` > git clone https://github.com/aws/aws-mwaa-local-runner.git > cd aws-mwaa-local-runner && ./mwaa-local-env build-image > paste the below content to `requirements/requirement.txt` --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt"; awscli==1.31.13 > ./mwaa-local-env test-requirements ``` ### Operating System docker ### Versions of Apache Airflow Providers ``` apache-airflow-providers-amazon==8.16.0 apache-airflow-providers-celery==3.5.1 apache-airflow-providers-cncf-kubernetes==7.13.0 apache-airflow-providers-common-io==1.2.0 apache-airflow-providers-common-sql==1.10.0 apache-airflow-providers-ftp==3.7.0 apache-airflow-providers-http==4.8.0 apache-airflow-providers-imap==3.5.0 apache-airflow-providers-mysql==5.10.0 apache-airflow-providers-sqlite==37.0 ``` ### Deployment Amazon (AWS) MWAA ### Deployment details _No response_ ### Anything else? awscli related packages: ``` botocore docutils colorama rsa ``` reducing the package version of docutils, colorama and rsa did the trick for me. ``` botocore==1.33.13 docutils==0.16 // docutils version supported by awscli 1.31.13 was docutils<0.17,>=0.10 colorama==0.4.4 // colorama version supported by awscli 1.31.13 was colorama<0.4.5 and >=0.2.5 rsa==4.7 // colorama version supported by awscli 1.31.13 was rsa<4.8 and >=3.1.2 ``` ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
