potiuk edited a comment on pull request #12703:
URL: https://github.com/apache/airflow/pull/12703#issuecomment-736070405
The problem is this line: "Installing remaining packages from 'all' extras"
We are running all remaining packages from "devel_all" extras: `pip install
-e ".[devel_all]"`. which installs "apache-beam" -but apparently this causes a
host of downgrades:
```
Attempting uninstall: cachetools
Found existing installation: cachetools 4.1.1
Uninstalling cachetools-4.1.1:
Successfully uninstalled cachetools-4.1.1
Attempting uninstall: httplib2
Found existing installation: httplib2 0.18.1
Uninstalling httplib2-0.18.1:
Successfully uninstalled httplib2-0.18.1
Attempting uninstall: google-resumable-media
Found existing installation: google-resumable-media 1.1.0
Uninstalling google-resumable-media-1.1.0:
Successfully uninstalled google-resumable-media-1.1.0
Attempting uninstall: pyarrow
Found existing installation: pyarrow 2.0.0
Uninstalling pyarrow-2.0.0:
Successfully uninstalled pyarrow-2.0.0
Attempting uninstall: mock
Found existing installation: mock 4.0.2
Uninstalling mock-4.0.2:
Successfully uninstalled mock-4.0.2
Attempting uninstall: google-cloud-bigquery-storage
Found existing installation: google-cloud-bigquery-storage 2.1.0
Uninstalling google-cloud-bigquery-storage-2.1.0:
Successfully uninstalled google-cloud-bigquery-storage-2.1.0
Attempting uninstall: google-cloud-bigquery
Found existing installation: google-cloud-bigquery 2.4.0
Uninstalling google-cloud-bigquery-2.4.0:
Successfully uninstalled google-cloud-bigquery-2.4.0
Attempting uninstall: fastavro
Found existing installation: fastavro 1.2.0
Uninstalling fastavro-1.2.0:
Successfully uninstalled fastavro-1.2.0
Attempting uninstall: dill
Found existing installation: dill 0.3.3
Uninstalling dill-0.3.3:
Successfully uninstalled dill-0.3.3
Attempting uninstall: google-cloud-vision
Found existing installation: google-cloud-vision 1.0.0
Uninstalling google-cloud-vision-1.0.0:
Successfully uninstalled google-cloud-vision-1.0.0
Attempting uninstall: google-cloud-videointelligence
Found existing installation: google-cloud-videointelligence 1.16.1
Uninstalling google-cloud-videointelligence-1.16.1:
Successfully uninstalled google-cloud-videointelligence-1.16.1
Attempting uninstall: google-cloud-spanner
Found existing installation: google-cloud-spanner 1.19.1
Uninstalling google-cloud-spanner-1.19.1:
Successfully uninstalled google-cloud-spanner-1.19.1
Attempting uninstall: google-cloud-pubsub
Found existing installation: google-cloud-pubsub 1.7.0
Uninstalling google-cloud-pubsub-1.7.0:
Successfully uninstalled google-cloud-pubsub-1.7.0
Attempting uninstall: google-cloud-dlp
Found existing installation: google-cloud-dlp 1.0.0
Uninstalling google-cloud-dlp-1.0.0:
Successfully uninstalled google-cloud-dlp-1.0.0
Attempting uninstall: google-cloud-bigtable
Found existing installation: google-cloud-bigtable 1.6.0
Uninstalling google-cloud-bigtable-1.6.0:
Successfully uninstalled google-cloud-bigtable-1.6.0
Attempting uninstall: google-cloud-storage
Found existing installation: google-cloud-storage 1.33.0
Uninstalling google-cloud-storage-1.33.0:
Successfully uninstalled google-cloud-storage-1.33.0
Attempting uninstall: apache-airflow
Found existing installation: apache-airflow 2.0.0b3
Uninstalling apache-airflow-2.0.0b3:
Successfully uninstalled apache-airflow-2.0.0b3
Running setup.py develop for apache-airflow
Successfully installed apache-airflow apache-beam-2.23.0
avro-python3-1.9.2.1 cachetools-3.1.1 crcmod-1.7 dill-0.3.1.1 fastavro-0.23.6
fasteners-0.15 google-apitools-0.5.31 google-cloud-bigquery-1.24.0
google-cloud-bigquery-storage-1.1.0 google-cloud-bigtable-1.0.0
google-cloud-datastore-1.7.4 google-cloud-dlp-0.13.0 google-cloud-pubsub-1.0.2
google-cloud-spanner-1.13.0 google-cloud-storage-1.29.0
google-cloud-videointelligence-1.13.0 google-cloud-vision-0.42.0
google-resumable-media-0.5.1 httplib2-0.17.4 mock-2.0.0 monotonic-1.5
oauth2client-3.0.0 pyarrow-0.17.1 pydot-1.4.1
````
Including `google-cloud-pubsub-1.0.2`
Apache-Beam is the only excluded package for CI image as it used to be known
by really bad requirements, forcing many packages to downgrade.
```
# Those packages are excluded because they break tests (downgrading mock)
and they are
# not needed to run our test suite.
PACKAGES_EXCLUDED_FOR_CI = [
'apache-beam',
]
```
We are however installing it for provider package check to be able to import
DataProc.
I was actually going to take a look at apache-beam for 2.0 now. I discussed
it today with @mik-laj that the recent 'apache-beam' has better requirements
and we should be able to install it without breaking too much - obviously not
that version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
