potiuk commented on pull request #14102:
URL: https://github.com/apache/airflow/pull/14102#issuecomment-774719469
This is only problem for 1.10.14 (backport packages) and 2.0.0 (regular
packages) build. The master build (where both master airflow and master
provider packages work fine).
The problem is coming from both airflow 2.0.0 and 1.10.14 having still the
old `azure-storage` as dependency in case of [snowflake] extra. Unfortunately
for the `azure` packages if the legacy package is mixed with the new ones the
`azure.storage.blob` and `azure.storeage.files` can be overwritten depending on
the installation process sequence. In our case - we have been doing this in the
CI image
1) Installing 'master` image from the CI with [all] extras
2) re-installing '1.10.14' airflow or '2.0.0' airflow with [all] extras and
providers from PyPI)
3) removing installed providers from PyPI
3) installing built providers from master without dependencies (assuming the
dependencies are already installed in step 1) - installing them with
dependencies is not a good idea because they might depend on unreleased yet
airflow version or aiflow providers that are missing in PyPI yet.
Unfortunately - in step 2 installing airflow 1.10.14 or 2.0.0 also installs
`azure-storage` legacy library which OVERRIDES the 'azure.storage.blob`,
`azure.storage.files` package -> this is the root cause of the failure. After
installing `azure-storage` the other azure libraries fail to import
Luckily this is easy to fix. I just run it in my master fork to be sure but
it seems to work well :) :
https://github.com/potiuk/airflow/runs/1849817483?check_suite_focus=true
The fix is to install airflow 1.10.14 / airflow 2.0.0 WITHOUT any extras in
step 2). This way we will keep dependencies installed in step 1) and no
`azure-storage` library will be ever installed (because it is only installed by
`snowflake` extra)
I make a PR to your PR shortly @kaxil - I will also move the depdencies to
`azure` extra in my PR (as in this case it will work perfectly fine).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
