mik-laj edited a comment on pull request #12188: URL: https://github.com/apache/airflow/pull/12188#issuecomment-749921653
> When I ran breeze locally, I found that azure-storage 0.36 is still getting installed Yes. We have extra protection to prevent image damage. In most cases, the constraints file is used to build the image, which contains a set of tested libraries. But we would like to update this file automatically sometimes, so we need the normal pip install to install the correct dependencies. More info: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/installation.html https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files Recently, snowflake-connector-python dropped support for old azure library, so now snowflake and azure can't be installed, because pip will have trouble finding dependencies. See: https://github.com/apache/airflow/pull/13195 In this PR, I am trying to improve compatibility with newer versions of Pinot and the pinot-python library, so I added the label "upgrade to newer dependencies" which means the image will be built without constraint, so the dependency installation may fail and it happened now. Several libraries have been updated and unfortunately, these dependencies do not work. Here I have compiled a list of the differences between installing with and without a constraints file. Most are safe and will be updated soon in constraints.txt as well, but we have a few changes we will add to setup.py/setup.cfg, so users will be limited in using these libraries. ```diff diff <(pip freeze | sort) <(curl https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt | sort) < -e /opt/airflow 19c18 < JPype1==1.2.0 --- > JPype1==1.1.2 30c29 < Pygments==2.7.3 --- > Pygments==2.7.2 33,35c32,34 < SQLAlchemy==1.3.22 < Sphinx==3.4.0 < Unidecode==1.1.2 --- > SQLAlchemy==1.3.20 > Sphinx==3.3.1 > Unidecode==1.1.1 56c55 < aws-sam-translator==1.33.0 --- > aws-sam-translator==1.31.0 58c57 < azure-batch==10.0.0 --- > azure-batch==9.0.0 65c64 < azure-keyvault-keys==4.3.1 --- > azure-keyvault-keys==4.3.0 90c89 < cachetools==4.2.0 --- > cachetools==4.1.1 94c93 < certifi==2020.12.5 --- > certifi==2020.11.8 97c96 < cfn-lint==0.44.1 --- > cfn-lint==0.42.0 109c108 < coverage==5.3.1 --- > coverage==5.3 111c110 < cryptography==3.3.1 --- > cryptography==3.2.1 113,114c112,113 < cx-Oracle==8.1.0 < dask==2020.12.0 --- > cx-Oracle==8.0.1 > dask==2.30.0 133c132 < eventlet==0.30.0 --- > eventlet==0.29.1 135,136c134,135 < facebook-business==9.0.1 < fastavro==1.2.1 --- > facebook-business==9.0.0 > fastavro==1.2.0 142c141,142 < flower==0.9.7 --- > flask-swagger==0.2.13 > flower==0.9.5 144c144,145 < fsspec==0.8.5 --- > fsspec==0.8.4 > funcsigs==1.0.2 152c153 < google-api-core==1.24.1 --- > google-api-core==1.23.0 156c157 < google-auth==1.24.0 --- > google-auth==1.23.0 160,161c161,162 < google-cloud-bigquery==2.6.1 < google-cloud-bigtable==1.6.1 --- > google-cloud-bigquery==2.4.0 > google-cloud-bigtable==1.6.0 163c164 < google-cloud-core==1.5.0 --- > google-cloud-core==1.4.3 178c179 < google-cloud-storage==1.35.0 --- > google-cloud-storage==1.33.0 184,185c185,186 < google-crc32c==1.1.0 < google-resumable-media==1.2.0 --- > google-crc32c==1.0.0 > google-resumable-media==1.1.0 191c192 < grpcio==1.34.0 --- > grpcio==1.33.2 196,197c197,198 < humanize==3.2.0 < hvac==0.10.6 --- > humanize==3.1.0 > hvac==0.10.5 212c213 < isort==5.6.4 --- > isort==4.3.21 219,220c220,221 < jsonpatch==1.28 < jsonpickle==1.4.2 --- > jsonpatch==1.27 > jsonpickle==1.4.1 232c233 < libcst==0.3.16 --- > libcst==0.3.14 237c238 < marshmallow==3.10.0 --- > marshmallow==3.9.1 239c240 < mock==4.0.3 --- > mock==4.0.2 245,246c246,247 < msal==1.8.0 < msgpack==1.0.2 --- > msal==1.6.0 > msgpack==1.0.0 250c251 < multidict==5.1.0 --- > multidict==5.0.2 267c268 < packaging==20.8 --- > packaging==20.4 269c270 < pandas==1.1.5 --- > pandas==1.1.4 281c282 < pinotdb==0.3.3 --- > pinotdb==0.1.1 285c286 < pre-commit==2.9.3 --- > pre-commit==2.9.2 290c291 < proto-plus==1.13.0 --- > proto-plus==1.11.0 292c293 < psutil==5.8.0 --- > psutil==5.7.3 296,297c297,298 < py==1.10.0 < pyOpenSSL==20.0.1 --- > py==1.9.0 > pyOpenSSL==19.1.0 306,307c307,308 < pydruid==0.6.2 < pyenchant==3.2.0 --- > pydruid==0.6.1 > pyenchant==3.1.1 311,312c312,313 < pylint==2.6.0 < pymongo==3.11.2 --- > pylint==2.5.3 > pymongo==3.11.1 324,325c325,326 < pytest-xdist==2.2.0 < pytest==6.2.1 --- > pytest-xdist==2.1.0 > pytest==6.1.2 355c356 < sendgrid==6.4.8 --- > sendgrid==6.4.7 357,358c358,359 < sentry-sdk==0.19.5 < setproctitle==1.2.1 --- > sentry-sdk==0.19.4 > setproctitle==1.2 364c365 < snowflake-connector-python==2.3.7 --- > snowflake-connector-python==2.3.6 367c368 < soupsieve==2.1 --- > soupsieve==2.0.1 391c392 < tableauserverclient==0.14.1 --- > tableauserverclient==0.14.0 396c397 < testfixtures==6.17.0 --- > testfixtures==6.15.0 404c405 < tqdm==4.54.1 --- > tqdm==4.54.0 414c415 < vertica-python==1.0.1 --- > vertica-python==1.0.0 416c417 < virtualenv==20.2.2 --- > virtualenv==20.2.1 424c425 < yandexcloud==0.64.0 --- > yandexcloud==0.61.0 ``` We probably wouldn't prioritize this, but we want updates for snowflake-connector-python as we have ordered a new update to this library that will include other critical changes that prevent the use of snowflake in Airflow 2.0. See: https://github.com/apache/airflow/issues/12881 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
